Merge pull request #238 from CLARIAH/dev

Merge updates from Dev for new release

.travis.yml

@@ -19,12 +19,14 @@ jobs:
     - stage: Tests
       os: osx
       language: objective-c
-      env: PYENV_VERSION=3.5.2
+      env: PYENV_VERSION=3.7.5
       # command to install dependencies
     - stage: Tests
       os: windows
       language: shell
-      env: PATH=/c/Python37:/c/Python37/Scripts:$PATH
+      env:
+        - PATH=/c/Python37:/c/Python37/Scripts:$PATH
+        - PYENV_VERSION=3.7.5
     - stage: deploy
       # deploy automatically to pypi
       before_deploy:

.travis/before_install.sh

@@ -8,8 +8,8 @@ if [[ $TRAVIS_BUILD_STAGE_NAME == 'Deploy' ]]; then
 fi
 
 if [[ $TRAVIS_OS_NAME == 'osx' ]]; then
-  brew install pyenv
-  pyenv install -s $PYENV_VERSION
+  brew install openssl pyenv
+  CONFIGURE_OPTS="--with-openssl=$(brew --prefix openssl)" pyenv install -s $PYENV_VERSION 
 elif [[ $TRAVIS_OS_NAME == 'windows' ]]; then
-  choco install python
+  choco install python --version $PYENV_VERSION
 fi

CITATION.cff

@@ -25,4 +25,4 @@ license: MIT
 message: "If you use this software, please cite it as below."
 repository-code: "https://github.com/CLARIAH/grlc"
 title: grlc
-version: "1.3.0"
+version: "1.3.1"

CONTRIBUTING.md

@@ -24,16 +24,25 @@ To get started with hacking grlc, follow these steps to create a local testing e
 1. `docker pull clariah/grlc:latest`
 2. `git clone https://github.com/CLARIAH/grlc`
 3. `cd grlc`
-4. `cp docker-compose.default.yml docker-compose.yml`
-5. Use your favorite editor to append these lines at the end of the file `docker-compose.yml` (replace `<GRLC_CLONE_PATH>` with the absolute path where you cloned grlc in step 2):
-
+4. Create a `docker-compose.yml` which matches your needs. For example:
 ```
-    volumes:
-     - <GRLC_CLONE_PATH>:/home/grlc/grlc
+version: '2'
+services:
+  grlc:
+    build: ./
+    restart: unless-stopped
+    ports:
+     - "8001:80"
+    environment:
+     - DEBUG=true
+     - USERMAP_GID=1000
+     - USERMAP_UID=1000
+     - GRLC_GITHUB_ACCESS_TOKEN=xxx
+     - GRLC_SERVER_NAME=grlc.io
 ```
 
-6. `docker-compose up`
-7. Your local grlc instance should be available at http://localhost:8001 and should respond to code modifications you make on `<GRLC_CLONE_PATH>`
+5. `docker-compose up`
+6. Your local grlc instance should be available at http://localhost:8001 and should respond to code modifications you make on `<GRLC_CLONE_PATH>`
 
 You're good to pick any issue at the  [issue tracker](https://github.com/CLARIAH/grlc/issues) marked as **enhancement** and start implementing it :)
 

README.md

@@ -43,69 +43,74 @@ For a quick usage tutorial check out our wiki walkthrough [here](https://github.
 
 ## Install and run
 
-Running via [docker](https://www.docker.com/) is the easiest and preferred form of deploying grlc. You'll need a working installation of [docker](https://www.docker.com/) and [docker-compose](https://docs.docker.com/compose/). To deploy grlc, just pull the latest image from Docker hub, and run docker compose with a `docker-compose.yml` that suits your needs (an [example](docker-compose.default.yml) is provided in the root directory):
+### grlc.io
+The easiest way to use grlc is by visiting [grlc.io/](http://grlc.io/) and using this service to convert SPARQL queries on your github repo into a RESTful API.
 
-<pre>
-git clone https://github.com/CLARIAH/grlc
-cd grlc
-docker pull clariah/grlc
-docker-compose -f docker-compose.default.yml up
-</pre>
-
-To run directly from Docker Hub it is sufficient to do:
-```
-docker run --rm -p 8088:80 -e GRLC_SERVER_NAME=grlc.io -e GRLC_GITHUB_ACCESS_TOKEN=xxx -e GRLC_SPARQL_ENDPOINT=http://dbpedia.org/sparql -e DEBUG=true clariah/grlc
-```
-
-(You can omit the first two commands if you just copy [this file](docker-compose.default.yml) somehwere in your filesystem)
-If you use the supplied `docker-compose.default.yml` your grlc instance will be available at http://localhost:8001
-
-If you want your grlc instance to forward queries to a different service than `grlc.io`, edit the `GRLC_SERVER_NAME` variable in your `docker-compose.yml` or `docker-compose.default.yml` file.
-
-In order for grlc to communicate with GitHub, you'll need to tell grlc what your access token is:
-
-1. Get a GitHub personal access token. In your GitHub's profile page, go to _Settings_, then _Developer settings_, _Personal access tokens_, and _Generate new token_
-2. You'll get an access token string, copy it and save it somewhere safe (GitHub won't let you see it again!)
-3. Edit your `docker-compose.yml` or `docker-compose.default.yml` file, and paste this token as value of the environment variable GRLC_GITHUB_ACCESS_TOKEN
-
-If you want to run grlc at system boot as a service, you can find example upstart scripts at [upstart/](upstart/grlc-docker.conf)
-
-### Alternative install methods
-
-Through these you'll miss some cool docker bundled features (like nginx-based caching). We provide these alternatives just for testing, development scenarios, or docker compatibility reasons.
+### Pip
+If you want to run grlc locally or use it as a library, you can install grlc on your machine. Grlc is [registered in PyPi](https://pypi.org/project/grlc/) so you can install it using pip.
 
 #### Prerequisites
 - Python3
 - development files:
-
 ```bash
 sudo apt-get install libevent-dev python-all-dev
 ```
 
-#### pip
-
-If you want to use grlc as a library, you'll find it useful to install via `pip`.
-
-<pre>
+#### pip install
+```bash
 pip install grlc
-grlc-server
-</pre>
+```
 
-More details can be found at [grlc's PyPi page](https://pypi.python.org/pypi/grlc) (thanks to [c-martinez](https://github.com/c-martinez)!).
+Grlc includes a command line tool which you can use to start your own grlc server:
+```bash
+grlc-server
+```
 
-#### Flask application
+#### Using gunicorn
+You can run grlc using gunicorn as follows:
+```bash
+gunicorn grlc.server:app
+```
 
-You can run grlc natively as follows:
+If you want to use your own gunicorn configuration, for example `gunicorn_config.py`:
+```
+workers = 5
+worker_class = 'gevent'
+bind = '0.0.0.0:8088'
 ```
-gunicorn -c gunicorn_config.py src.server:app
+Then you can run it as:
+```bash
+gunicorn -c gunicorn_config.py grlc.server:app
 ```
 
 **Note:** Since `gunicorn` does not work under Windows, you can use `waitress` instead:
+```bash
+waitress-serve --port=8088 grlc.server:app
 ```
-waitress-serve --port=8088 src.server:app
+
+#### Grlc library
+You can use grlc as a library directly from your own python script. See the [usage example](https://github.com/CLARIAH/grlc/blob/master/doc/notebooks/GrlcFromNotebook.ipynb) to find out more.
+
+### Docker
+To run grlc via [docker](https://www.docker.com/), you'll need a working installation of docker. To deploy grlc, just pull the [latest image from Docker hub](https://hub.docker.com/r/clariah/grlc/). :
+```bash
+docker run -it --rm -p 8088:80 clariah/grlc
 ```
 
-You can also find an example [here](https://github.com/CLARIAH/grlc/blob/master/docker-assets/entrypoint.sh#L24)
+The docker image allows you to setup several environment variable such as `GRLC_SERVER_NAME` `GRLC_GITHUB_ACCESS_TOKEN` and `GRLC_SPARQL_ENDPOINT`:
+```bash
+docker run -it --rm -p 8088:80 -e GRLC_SERVER_NAME=grlc.io -e GRLC_GITHUB_ACCESS_TOKEN=xxx -e GRLC_SPARQL_ENDPOINT=http://dbpedia.org/sparql -e DEBUG=true clariah/grlc
+```
+
+## Access token
+
+In order for grlc to communicate with GitHub, you'll need to tell grlc what your access token is:
+
+1. Get a GitHub personal access token. In your GitHub's profile page, go to _Settings_, then _Developer settings_, _Personal access tokens_, and _Generate new token_
+2. You'll get an access token string, copy it and save it somewhere safe (GitHub won't let you see it again!)
+3. Edit your `docker-compose.yml` or `docker-compose.default.yml` file, and paste this token as value of the environment variable GRLC_GITHUB_ACCESS_TOKEN
+
+If you want to run grlc at system boot as a service, you can find example upstart scripts at [upstart/](upstart/grlc-docker.conf)
 
 ## Usage
 

bin/grlc-server

@@ -1,15 +1,24 @@
-#!/home/amp/src/grlc/bin/python2
+#!/usr/bin/env python
+"""Grlc server.
 
+Usage:
+  grlc-server [--port=PORT]
+
+Options:
+  --port=PORT   Port the server runs on [default: 8088].
+"""
+from docopt import docopt
 from grlc.server import app as grlc_app
+from grlc import __version__ as grlc_version
 from grlc import static
 from sys import platform
 
 
-def runViaWaitress():
+def runViaWaitress(port=8088):
     from waitress import serve
-    serve(grlc_app, listen='*:8088')
+    serve(grlc_app, listen='*:%d'%port)
 
-def runViaGunicorn():
+def runViaGunicorn(port=8088):
     from gunicorn.app.base import BaseApplication
     from gunicorn.six import iteritems
     class StandaloneApplication(BaseApplication):
@@ -28,14 +37,17 @@ def runViaGunicorn():
             return self.application
 
     options = {
-        'bind': '%s:%d' % ('0.0.0.0', 8088),
+        'bind': '%s:%d' % ('0.0.0.0', port),
         'workers': 20,
         'debug': static.LOG_DEBUG_MODE
     }
     StandaloneApplication(grlc_app, options).run()
 
 if __name__ == '__main__':
+    args = docopt(__doc__, version='Grlc %s server'%grlc_version)
+    port = int(args['--port'])
+
     if platform=='win32':
-        runViaWaitress()
+        runViaWaitress(port)
     else:
-        runViaGunicorn()
+        runViaGunicorn(port)

config.default.ini

@@ -12,4 +12,4 @@ server_name = grlc.io
 user = none
 password = none
 # Logging level
-debug = True
+debug = False

docker-assets/entrypoint.sh

@@ -25,7 +25,7 @@ case ${1} in
         # configure default sparql endpoint
         sed -i "s|http://dbpedia.org/sparql|${GRLC_SPARQL_ENDPOINT}|" config.ini
 
-        gunicorn -c gunicorn_config.py src.server:app
+        grlc-server --port=8088
         # migrate_database
         # rm -rf /var/run/supervisor.sock
         # exec /usr/bin/supervisord -nc /etc/supervisor/supervisord.conf

requirements-test.txt

@@ -1,2 +1,2 @@
 mock==2.0.0
-pytest==3.4.2
+pytest==5.2.1

requirements.txt

@@ -1,3 +1,4 @@
+docopt==0.6.2
 Flask==1.0.2
 Flask-Cors==3.0.6
 gevent==1.4.0
@@ -16,10 +17,9 @@ requests==2.20.0
 six==1.11.0
 simplejson==3.16.0
 setuptools>=38.6.0
-SPARQLTransformer==1.6.10
+SPARQLTransformer==1.7.0
 SPARQLWrapper==1.8.2
-Werkzeug==0.15.3
+werkzeug>=0.16.0
 PyGithub==1.43.5
-pythonql3==0.9.61
 gunicorn==19.6.0; sys_platform!="win32"
-waitress==1.1.0; sys_platform=="win32"
+waitress==1.4.0; sys_platform=="win32"

setup.py

@@ -13,7 +13,7 @@
         root_dir = root.replace(grlc_base_dir, '')
         data_files = os.path.join(root_dir, '*')
         grlc_data.append(data_files)
-grlc_version = '1.3.0'
+grlc_version = '1.3.1'
 
 with codecs.open('requirements.txt', mode='r') as f:
     install_requires = f.read().splitlines()

src/__init__.py

@@ -1 +1 @@
-__version__ = '1.3.0'
+__version__ = '1.3.1'

src/cache.py

@@ -3,12 +3,13 @@
 # cache.py: grlc spec caching utilities
 import json
 import urllib.request, urllib.error, urllib.parse
-import logging
+import grlc.glogging as glogging
+
+glogger = glogging.getGrlcLogger(__name__)
 
 # Name of the cache json file
 CACHE_NAME = "db-cache.json"
 
-glogger = logging.getLogger(__name__)
 def init_cache():
     '''
     Initializes the grlc cache (json file)