|
1 | | -# TIL v2 |
| 1 | +# Today I Learned (TiL) - v2 |
2 | 2 |
|
3 | | -Today I Learned is back 😎 |
| 3 | +TIL is a simple software that aim to enable article and reflexion-sharing across all Zenika agencies in the world. It |
| 4 | +acts as an aggregator (as |
| 5 | +daily.dev can do), but on-premises and with filters on content to avoid being spammed with news that don't interest us. |
4 | 6 |
|
5 | | -Documentation will come shortly |
6 | 7 |
|
7 | | -**THIS IS A WIP!** |
| 8 | +## Getting started |
8 | 9 |
|
9 | | -Run it: `TIL_SERVER_PORT=9023 TIL_DATABASE_FILE_NAME=toto.db TIL_DEBUG=false TIL_GOOGLE_CLIENT_ID=my_id TIL_GOOGLE_CLIENT_SECRET=my_secret TIL_JWT_SECRET=aaaaaa go run -tags=viper_bind_struct ./cmd` |
| 10 | +### Developer tools |
10 | 11 |
|
11 | | -Proudly powered by the Dobby Team. |
| 12 | +If you would like to develop TiL, the best way is to simply install Go on your computer ; then run |
| 13 | +`go run -tags=viper_bind_struct ./cmd`. The server |
| 14 | +will start on port 8000. |
| 15 | + |
| 16 | +### Docker |
| 17 | + |
| 18 | +If you simply want to run the application, Docker is the best way to do. Simply run `docker build -t til:latest .`, and |
| 19 | +run it! Full configuration |
| 20 | +reference below. Please think to mount database file in your container if you want persistent data! |
| 21 | + |
| 22 | +## Configuration |
| 23 | + |
| 24 | +All the configuration is done through environment variables. Viper is in charge of automatically parse them and load |
| 25 | +them in our Go application. The |
| 26 | +available environment variables are the following: |
| 27 | + |
| 28 | +* **(Mandatory)** `TIL_JWT_SECRET`: The secret key used to sign JWT tokens issued by the app. |
| 29 | +* **(Mandatory)** `TIL_GOOGLE_CLIENT_ID`: The Client ID for Google Authentication |
| 30 | +* **(Mandatory)** `TIL_GOOGLE_CLIENT_SECRET`: The Client Secret for Google Authentication |
| 31 | +* *(Optional)* `TIL_DEBUG` (default: `false`): Set this variable to `true` if you want to enable debug logs on |
| 32 | + application. Please be careful: this mode will log WAY MORE information, even secret tokens! The debug mode will also |
| 33 | + disable token expiry time check, which may lead to a lot of JWT renewal. Do not use this in production. |
| 34 | +* *(Optional)* `TIL_DATABASE_FILE_NAME` (default: `til.sqlite3`): Specify database file name (and location). |
| 35 | +* *(Optional)* `TIL_SERVER_PORT` (default: `8000`): Server port to listen on |
| 36 | +* *(Optional)* `TIL_GOOGLE_TOKEN_ENDPOINT` (default: `https://oauth2.googleapis.com/token`): The Google Authentication |
| 37 | + endpoint to validate user token |
| 38 | +* *(Optional)* `TIL_DEFAULT_ADMIN` (default: none): The user to immediately promote as admin. Use his Google ID in this |
| 39 | + variable to make it a superuser of the application. This variable will only take effect if ALL the following conditions are met : |
| 40 | + * The variable is set to a non-void value; |
| 41 | + * There is no other user (admin or not) in the database. |
| 42 | + |
| 43 | + |
| 44 | +## Tests |
| 45 | + |
| 46 | +Tests are made using Gherkin (Cucumber). This allows non-technical people to understand and collaborate to tests, |
| 47 | +improving security and coverage. |
| 48 | + |
| 49 | +All the tests can be find in `test/` folder. We are using a mock for the Google Authentication Server so you'll never |
| 50 | +have to provide a secret to run tests. The `test/` folder includes some secrets (notably an RSA key and a JWT secret |
| 51 | +key): these secrets are dedicated to our tests, and have never been used in production. Please do not report them as a |
| 52 | +security issue; we're perfectly aware of it. Obviously, do not use these secrets for your own production (otherwise, |
| 53 | +magic will probably happen!). |
| 54 | + |
| 55 | +All our tests are containerized to avoid configuration issues. To run them, simply type `go test -v ./test/`, and Golang |
| 56 | +will do the rest. |
| 57 | + |
| 58 | +The following sentences are available : |
| 59 | + |
| 60 | +#### Authentication |
| 61 | + |
| 62 | +* `I have a JWT token for my regular|admin user` |
| 63 | + * This call will connect a user with the defined privilege. Every subsequent API calls will be authenticated using |
| 64 | + this JWT token. |
| 65 | +* `I clear the current JWT token` |
| 66 | + * Remove the current JWT token (if any). Every subsequent API calls will be unauthenticated. |
| 67 | + |
| 68 | +#### Database |
| 69 | + |
| 70 | +* `I reset the database` |
| 71 | + * Reset the database by restarting the container |
| 72 | + |
| 73 | +#### HTTP |
| 74 | + |
| 75 | +* `I send a "GET|POST|PUT|DELETE" request to "ENDPOINT"` |
| 76 | + * Send a request to the defined endpoint (an endpoint is an API path, for example `/posts`). |
| 77 | +* `I send a "GET|POST|PUT|DELETE" request to "ENDPOINT" with payload` |
| 78 | + * Same as before, but add a payload to the request. The payload must be placed on the next line, between `"""`. See |
| 79 | + example below or in `test/features/test.feature` for more details. |
| 80 | +* `the response code should be XXX` |
| 81 | + * Check that the previous request sent have the defined status code. |
| 82 | +* `the response should have XXX items in path "PATH"` |
| 83 | + * **WORKS ONLY ON ARRAYS** Count the number of items returned by the server in the JSON file. |
| 84 | + * The path is the place on the JSON to look, dot-separated. `@` is the main document. For example, if you want to |
| 85 | + access the `items` part of your JSON, the path will be `@.items`. |
| 86 | +* `I save the "XXXXX" header as "XXXXXX" for suite` |
| 87 | + * Save the content of a header in a variable for later re-use. For example, |
| 88 | + `I save the "X-Id" header as "postId" for suite` will allow you to write |
| 89 | + `I send a "GET" request to "/posts/{{postId}}"` later on. |
| 90 | +* `I save the value "XXX" in path "YYY" as "ZZZ" for suite` |
| 91 | + * Save the value XXX located in path YYY in a variable for later reuse. For example, `I save the value "id" in path "@" as "userId" for suite` |
| 92 | +* `the response should have the following content in path "XXX"` |
| 93 | + * **WORKS ONLY ON OBJECTS** Check that the specified values for each key are correct. |
| 94 | +* `the response should have the following items in path "XXX"` |
| 95 | + * **WORKS ONLY ON ARRAYS** Same as the previous one, but for arrays. |
| 96 | +* `the response should have the key "XXX" in path "YYY"` |
| 97 | + * Check if the response contains a specified key in the desired path. For example `the response should have the key "id" in path "@.items.0"` |
| 98 | +* `the response should not have the key "XXX" in path "YYY"` |
| 99 | + * The exact opposite as before ;-) |
| 100 | + |
| 101 | + |
| 102 | +### Example |
| 103 | + |
| 104 | +In this example, we log in as a regular user, then create a post before accessing it. |
| 105 | + |
| 106 | +```gherkin |
| 107 | +Feature: Demo |
| 108 | +
|
| 109 | + Scenario: Demo |
| 110 | + When I have a JWT token for my regular user |
| 111 | + And I send a "POST" request to "/posts" with payload |
| 112 | + """ |
| 113 | + { |
| 114 | + "title": "Test", |
| 115 | + "link": "https://lamontagne.fr", |
| 116 | + "tags": ["lang:fr"] |
| 117 | + } |
| 118 | + """ |
| 119 | + And the response code should be 201 |
| 120 | + And I save the "X-Post-Id" header as "postId" for suite |
| 121 | + And I send a "GET" request to "/posts/" |
| 122 | + And the response code should be 200 |
| 123 | + And the response should have the following content in path "@" |
| 124 | + | total_items | total_pages | current_page | items_per_page | |
| 125 | + | 1 | 1 | 0 | 20 | |
| 126 | + And the response should have the following items in path "@.items" |
| 127 | + | id | title | link | |
| 128 | + | {{postId}} | Test | https://lamontagne.fr | |
| 129 | + And the response should have 1 items in path "@.items.0.tags" |
| 130 | + And the response should have the following items in path "@.items.0.tags" |
| 131 | + | lang:fr | |
| 132 | +``` |
| 133 | + |
| 134 | +## Will it scale? |
| 135 | + |
| 136 | +TiL is conceived following the KISS way of life; but it doesn't mean it won't scale. During our tests, we ran the |
| 137 | +following Locust file, with 50 concurrent users running 131 RPS: |
| 138 | + |
| 139 | +```py |
| 140 | +from locust import HttpUser, between, task |
| 141 | + |
| 142 | + |
| 143 | +class WebsiteUser(HttpUser): |
| 144 | + @task |
| 145 | + def index(self): |
| 146 | + self.client.get("/posts", headers={"Authorization": "token"}) |
| 147 | + |
| 148 | + |
| 149 | + @task |
| 150 | + def publish(self): |
| 151 | + self.client.post("/posts", headers={"Authorization": ""}, json={"title":"essai", "link":"https://www.lamontagne.fr/paris-75000/actualites/coupure-de-courant-geante-habitants-dans-la-rue-metros-et-avions-a-l-arret-les-images-du-chaos-en-espagne-et-au-portugal_14679007/", "tags": ["failure", "electricity", "lang:fr"], "content": "severe electricty shortage in Spain and Portugal"}) |
| 152 | +``` |
| 153 | + |
| 154 | +During 10 minutes, the locust file generated roughly 30k entries in database, and the same amount of article queried. |
| 155 | +The Locust file ended with 0 failure and a MTRR around 600ms, which is a good result stating that our test is far above |
| 156 | +the real frequentation. |
0 commit comments