Persistent (#111)

* first pass on scheduled queries

* wire query scheduler into the app

* next pass

* next pass

* next pass

* loading sessions

* apache drill and s3

* 🐨 update

* next pass

* rewrite front-end

* apache spark and s3 previews

* ..

* next pass

* fix UI and put tab components into separate files

* heroku

* heroku cleanup

* typo

* rbuild

* log down node version

* case sensitive

* fix urls for headless and app mode

* protocol typo

* rm secretAccessKey from API

* update todo

* link shim

* pre-save check for persistence

* first pass at social auth

* Sets table previews as a separate component

* Fixes fixed widths for user inputs

* Adds an alert window for deleting a tab

* Rewrites external links to support href in browser & electron

* 🐄

* Fixes deeleting credentials

* Fix uncaught rebase issue

* move elasticsearch to raw API and update tests

* Merge

* minor

* on-prem fixes

* fix url

* Adds fun tables() to /connections/elasticsearch.js

* Elasticsearch index and docs dropdown

* Irons out elasticsearch previews

* Changed all files' format: JSON -> YAML

* include "Unauthenticated" in the error message

* return 200 so that we can return json

* fix elasticsearch tests

* Putting back https part 1

* update elasticsearch

* replace `selectedIndex` with `index`, `selectedType` with `type`

* 🔧 replace `refreshRate` with `refreshInterval`

* 🔧 set refreshInterval as seconds not milliseconds

* update rate to interval error message

* Remove dead code

* Change default port from 9494 to 9000

* Ignore log files

* Putting back https part 2

* Index 0 -> R.head

* putting back https part 3

* putting back https part 4

* finish cleaning dead code

* fix and test oauth2 endpoint

* add watchers

* remove .only from tests

* sanity test home page route

* Improve OAuth interface

* note about error messages

* another TODO

* note about yaml

* prefix ENV variables with `PLOTLY_CONNECTOR`

* ⚡

* Replace "credential" with "connection"

* Revert "Replace "credential" with "connection""

This reverts commit 7c1e6fc.

* Replace `credential` with `connection`

* ignore generated stylesheet

* 🙈 Except that `credentials` can sometimes be singular...

Replace a bunch of instances of `credentials` with `connection` since
we were previously referring to `credentials` as a single object with a
few parameters. This is now just a `connection`.
Also, we were using `Connections.js` to refer to the different
Datastore backends. Rename this to a Datastore.js module.

* Another credentials -> connections file rename

* protect the `electron` import - can't use `import require` in browser targets

* ⚡ dead code

* remove some `console.log`s

* `getCerts` is breaking non-electron builds for some reason

comment it out for now

* comment about cert file locations

* remove elasticsearch `index` from test fixtures

* Perform a pre-query authentication check

* Fix deleting queries

* Fix the format of tables

should be `[‘table 1’, ‘table 2’]` not `[[‘table 1’], [‘table 2’]]`

* Update names of backend routes

mostly for all actions to fall under `/connections`

* Fix an `electron` import to work in node environments and tests

* remove nrows and ncols

* fix and update `routes.spec.js` tests. all 29 are passing now

* add unit testing npm command

usage:
```
$ npm run test-unit test/backend/persistent/routes.spec.js --watch
```

* update front-end with new endpoints

* include the dialect in the connection ID

other contexts like shareplot will only have the connectionID, they
won’t have the dialect. storing the dialect as part of the connectionID
will allow those contexts to render dialect-specific UIs without making
an additional request to the connector

* Remap heroku's $PORT to our $PLOTLY_CONNECTOR_PORT on heroku build

* set heroku's port in the procfile

* Fix more tests. 47/50 are passing now.

* 🐨 Edit credentials button

* 🛀 and update todo.md

* move https to front end

* fix dynamic import

* 3 SQL fixes

- Redshift tables
- SQLite connect
- Tables across all dialects

* Rename `Connections` to `Datastores` in routes

* Check that connections are valid before saving them

* Remove `ncols`, `nrows`

* 📖

* ⚡

* Add tests for new "connect before save" behaviour and test every dialect

* 🍤 lock in dependencies with shrinkwrap

* ⚡

* new access token

gotta figure out why these are expiring

* Attempt to fix linux build error

Builds on heroku were failing due to:

This thread: npm/npm#14042 recommended
switching to `3.10.10` and adding `fsevents: ‘*’` in optional deps.

Got to switch to Yarn soon!

* attempt downgrading npm to fix heroku shrinkwrap build issue

* Try removing postinstall

* rm shrinkwrap until yarn - too many cross-platform errors

* improve refreshInterval error message

* bump version of npm

as recommended here: github.com/npm/npm/releases

* We can't use `fs` in the web-app front end

comment out `https` so that it works in the web browser for now

* comment out httpsUtils since it doesn't work in non-electron envs

* try to get shrinkwrap working again

* finally fix https

* 🛀 and fix web-build

* 🐨 edit credentials part 1

* 🔧 split error in the web build for __filename

* Change `PLOTLY_API_DOMAIN` to `PLOTLY_API_URL`

* ⚡

* Docker updates:

- Add support for custom API domains
- Add support for data mapping between host and container
- Add support for shrinkwrap
- Log to stdout

* typo

* TODO about file location and logs

* Didn't mean to commit this

* ⭐ edit credentials

* 🐯 for `PUT credentials/:id`

* 🐨 for section titles

* ⚡  console warns and some other 🛀

* update TODO

* 🔧 weird state occurence when updating crews

* Another `CREDENTIALS` -> `CONNECTIONS` change

* Make the storage folder configurable

Also fix a few tests

* update a few comments

* add todos about untested endpoints

* Include sample connections in the repo so that heroku can use them

* remove generated style folder

* Include the storage path env variable in the Procfile

This allows the heroku instance to read the settings from the example
databases included in this repo in the `sample-storage` folder

* Include an empty logfile to write to

* heroku directory is `app` not `web`

* nice big favicons

* rm sample log file

* fix typo in connections and add postgres

* move sqlite into the group of sql-like connections

* update-credentials: add tests and validate

* 🔧 the hack in tests to start an http server

* update credentials: UX/UI fixes

* Support non-owners updating queries

* update expected results from elasticsearch test database

* 🙈 typo

* include domain as runtime env variable instead of build-time env variable

* fix error handling

* remove blank settings

* ⭐ scrolling

* do not package certs into the release

* 🙈 requestor

* use a 200K rows database for scroll tests

* make the SSL option clearer

* 🙈 was calling `clearInterval` on the value, not the key

* this is not an error - it can happen on regular usage

* clean up logs

* fixup 404 behaviour

* fix elasticsearch tests with the new elasticsearch data

* fix tests

* fix more tests

* fix more tests

* Wait no - I had it right. And now I have the tests to prove it.

* update comment

* handle quotes and comma escapes in CSV parsing

* Even if the owner is the requestor, check that they have write permissions

* Update error message for front end

* fix tests

* Keep consistent order of columns in elasticsearch queries

* Update tests with new column order

* use a new index to test scroll that has linear values

and update tests so that we’re testing the actual values, not just the
lengths

* remove .only

* A new Apache Drill test instance with more memory

* support elasticsearch `geo_point`

* fix tests with new remote datastore configs

* apache drill error messages and tests

* 🐨 fixes for options

* ignore all of those log files

* external button links to workspace (onprem or normal)

* 🐨 for https step when setup

* 🐨 fix for buttons style to match

* options checkbox bad position

* 🛀 cleanup TODO.md and Configuration.react.js

* fixes domain error

* use relative URLs for static content

* Fix dockerfile comment (thanks @scjody !)

* change some copy to test

* Remove test element

* don't log to file if we're only logging to `stdout`

* typo 🙈

* oauth URL fixes

* display dev tools even in production

otherwise it’s really hard to help our users debug what might be going
wrong

* fix dockerfile mapping

* bump version to 1.0.0

* `build-web` when building the package too

at the least for oauth!

* rm elasticsearch from package

lots of other unused packages in here too

* try adding `csv-parse` to the external list to get builds to work

* update which ways the arrows point to indicate expanded or collapsed

`>` commonly represents collapsed, and the down one usually represents
collapsed.

* remove upgrade button from on-prem

* change CTA to plans

* fixes workspace button

* swap e2e tests with unit tests on circle

* linux is case sensitive

* fix some tests

* lint

* Fix error message

* 🙈 typo

* make tests less fragile

* `mkdirp.sync`

* rm lint from tests for now and define concat

* increase the timeouts for circle

* 🙈 `this`

* fix https section for windows

* fix oauth2 urls for on-prem

* semver our api

* show error handling when connect aborts after save error

* 🙈

* cleanup todos and dead code

* fixes elasticsearch previews

* comments on handling saving errors

* table cell can be a dict

* missed a couple of conflicts

Author: chriddyp

.dockerignore

@@ -0,0 +1,49 @@
+# Logs
+logs
+*.log
+
+# Runtime data
+pids
+*.pid
+*.seed
+
+# Directory for instrumented libs generated by jscoverage/JSCover
+lib-cov
+
+# Coverage directory used by tools like istanbul
+coverage
+
+# Grunt intermediate storage (http://gruntjs.com/creating-plugins#storing-task-files)
+.grunt
+
+# node-waf configuration
+.lock-wscript
+
+# Compiled binary addons (http://nodejs.org/api/addons.html)
+build/Release
+
+# Dependency directory
+node_modules
+
+# OSX
+.DS_Store
+
+# App packaged
+dist
+release
+main.js
+main.js.map
+static/*.js*
+
+# Local tests
+test/set_creds.sh
+
+*.pem
+
+*.srl
+
+.srl
+
+backend/headlessBundle.js
+
+backend/headlessBundle.js.map

.gitignore

@@ -1,12 +1,11 @@
-# Logs
-logs
-*.log
-*.log.*
-
+npm-debug.log
 # Runtime data
 pids
 *.pid
 *.seed
+sample-storage/log.log
+*.log
+*.log.*
 
 # Directory for instrumented libs generated by jscoverage/JSCover
 lib-cov
@@ -35,6 +34,8 @@ dist
 release
 main.js
 main.js.map
+static/*.js*
+static/style.css
 
 # Local tests
 test/set_creds.sh
@@ -47,3 +48,7 @@ backend/headlessBundle.js.map
 *.srl
 
 .srl
+
+backend/headlessBundle.js
+
+backend/headlessBundle.js.map

Dockerfile

@@ -1,2 +1,45 @@
-# 
+# Usage:
+# $ PLOTLY_API_DOMAIN="api-local.plot.ly" PLOTLY_STREAMBED_SSL_ENABLED="true" docker build -t chriddyp/database-connector .
+# $ docker run -p 9494:9494 -e PLOTLY_CONNECTOR_PLOTLY_API_DOMAIN=$PLOTLY_DOMAIN_API -e PLOTLY_CONNECTOR_PLOTLY_API_SSL_ENABLED=$PLOTLY_STREAMBED_SSL_ENABLED -d chriddyp/database-connector -v ~/.plotly/connector:/home
+#
+# Depends on the following ENV variables:
+# - PLOTLY_DOMAIN_API (e.g. api-local.plot.ly)
+# - PLOTLY_STREAMBED_SSL_ENABLED (e.g. true)
+# Adapted from https://nodejs.org/en/docs/guides/nodejs-docker-webapp/
 FROM node:6.6
+
+# I'd like to install the latest version of npm with something like this:
+# RUN npm install --global [email protected]
+# But unforunately that fails with
+# It doesn't seem like there is a good solution in the community yet
+# See: https://github.com/npm/npm/issues/9863
+
+# The App saves folders to the `os.homedir()` directory.
+ENV HOME=/home/
+
+# Create app directory
+RUN mkdir -p /usr/src/app
+WORKDIR /usr/src/app
+
+# Create a directory that the app can save files to and that the host can access
+# To map to the files that are saved on your host, run with:
+# $ docker run -v ~/.plotly/connector:/plotly-connector
+# TODO - With that command ^^, it's redudant to call `VOLUME /plotly-connector` right?
+# Save that directory as an ENV variable for the app to use when saving files
+# By default, the app will save these files in the home directory
+ENV PLOTLY_CONNECTOR_DATA_FOLDER="/plotly-connector"
+
+# Log to stdout in addition to the file
+ENV PLOTLY_CONNECTOR_LOG_TO_STDOUT="true"
+
+# Install app dependencies
+COPY package.json /usr/src/app
+COPY npm-shrinkwrap.json /usr/src/app
+RUN npm install --no-optional
+
+COPY . /usr/src/app
+RUN npm run heroku-postbuild
+
+ENV PLOTLY_CONNECTOR_PORT 9494
+EXPOSE 9494
+CMD [ "node", "/usr/src/app/dist/headless-bundle.js" ]

Procfile

@@ -1 +1 @@
-web: node ./backend/headlessBundle.js
+web: PLOTLY_CONNECTOR_PORT=$PORT PLOTLY_CONNECTOR_STORAGE_PATH="/app/sample-storage" node ./dist/headless-bundle.js

README.md

@@ -47,7 +47,7 @@ $ npm run dev
 
 ### Testing
 
-Note: Currently access to remote databases is required to run local tests. These credentials are not committed to git at the moment. Contact us if you require running tests locally.
+Note: Currently access to remote databases is required to run local tests. These connections are not committed to git at the moment. Contact us if you require running tests locally.
 
 ```bash
 $ npm run build

TODO.md

@@ -0,0 +1,23 @@
+- [x] Parquet files hang sometimes
+- [x] postgis instance
+- [ ] fail when registering queries
+- [ ] whats with that "Can't set headers after request is sent" error message??
+- [ ] clean up all of the Logger.log calls - the whitespace that the back tick quotes introduce is real annoying.
+- [ ] elasticsearch UI on prod
+- [ ] run in a docker container
+- [ ] publish architecture
+- [ ] fix documentation links
+
+- [ ] start running against prod
+
+little things
+- update e2e tests
+    - test that server is running (keep majority of api tests in routes.spec.js)
+    - test headless mode
+- update `updateGrid` in QueryScheduler to remove query if the response was a 404
+
+***
+
+- Tell people about the logs in the UI
+- Need to update shareplot and workspace dates to be "date modified" instead of
+  date created.