en

Author: forsaken628

docs/en/sql-reference/20-sql-functions/07-aggregate-functions/aggregate-markov-train.md

@@ -0,0 +1,49 @@
+---
+title: MARKOV_TRAIN
+---
+
+Extracting patterns from datasets using Markov models
+
+## Syntax
+
+```sql
+MARKOV_TRAIN(<string>)
+
+MARKOV_TRAIN(<order>)(<string>)
+
+MARKOV_TRAIN(<order>, <frequency_cutoff>, <num_buckets_cutoff>, <frequency_add>, <frequency_desaturate>) (<string>)
+```
+
+## Arguments
+
+| Arguments | Description |
+|------------------| ------------------ |
+| `string` | Input |
+| `order` | Order of markov model to generate strings |
+| `frequency-cutoff` | Frequency cutoff for markov model: remove all buckets with count less than specified |
+| `num-buckets-cutoff` | Cutoff for number of different possible continuations for a context: remove all histograms with less than specified number of buckets |
+| `frequency-add` | Add a constant to every count to lower probability distribution skew |
+| `frequency-desaturate` | 0..1 - move every frequency towards average to lower probability distribution skew |
+
+## Return Type
+
+Depending on the implementation, it is only used as a argument for [MARKOV_GENERATE](../20-other-functions/markov_generate.md).
+
+## Examples
+
+```sql
+create table model as
+select markov_train(concat('bar', number::string)) as bar from numbers(100);
+
+select markov_generate(bar,'{"order":5,"sliding_window_size":8}', 151, (number+100000)::string) as generate
+from numbers(5), model;
++-----------+
+| generate  |
++-----------+
+│ bar95     │
+│ bar64     │
+│ bar85     │
+│ bar56     │
+│ bar95     │
++-----------+
+```

docs/en/sql-reference/20-sql-functions/07-aggregate-functions/index.md

@@ -88,4 +88,10 @@ This page provides a comprehensive overview of aggregate functions in Databend,
 | Function | Description | Example |
 |----------|-------------|---------|
 | [RETENTION](aggregate-retention.md) | Calculates retention rates | `RETENTION(action = 'signup', action = 'purchase')` → `[100, 40]` |
-| [WINDOWFUNNEL](aggregate-windowfunnel.md) | Searches for event sequences within time window | `WINDOWFUNNEL(1800)(timestamp, event='view', event='click', event='purchase')` → `2` |
+| [WINDOWFUNNEL](aggregate-windowfunnel.md) | Searches for event sequences within time window | `WINDOWFUNNEL(1800)(timestamp, event='view', event='click', event='purchase')` → `2` |
+
+## Anonymization
+
+| Function | Description | Example |
+|----------|-------------|---------|
+| [MARKOV_TRAIN](aggregate-markov-train.md) | train markov model | `MARKOV_TRAIN(address)` |

docs/en/sql-reference/20-sql-functions/17-table-functions/index.md

@@ -52,3 +52,9 @@ This page provides reference information for the table functions in Databend. Ta
 |----------|-------------|--------|
 | [ICEBERG_MANIFEST](./iceberg-manifest.md) | Shows Iceberg table manifest information | `SELECT * FROM ICEBERG_MANIFEST('mytable')` |
 | [ICEBERG_SNAPSHOT](./iceberg-snapshot.md) | Shows Iceberg table snapshot information | `SELECT * FROM ICEBERG_SNAPSHOT('mytable')` |
+
+## Anonymization
+
+| Function | Description | Example |
+|----------|-------------|---------|
+| [OBFUSCATE](obfuscate.md) | dataset anonymization | `SELECT * FROM OBFUSCATE(users)` |

docs/en/sql-reference/20-sql-functions/17-table-functions/obfuscate.md

@@ -0,0 +1,72 @@
+---
+title: OBFUSCATE
+---
+
+Dataset anonymization. This is a quick tool, and for more complex scenarios, it is recommended to directly use the underlying function [MARKOV_TRAIN](../07-aggregate-functions/aggregate-markov-train.md), [MARKOV_GENERATE](../20-other-functions/markov_generate.md), [FEISTEL_OBFUSCATE](../20-other-functions/feistel_obfuscate.md).
+
+## Syntax
+
+```sql
+OBFUSCATE('<table>'[, seed => <seed>])
+```
+
+## Examples
+
+```sql
+create or replace table users as 
+select * from (values 
+(1, 'James Smith', '[email protected]', '123 Fake St, Anytown, CA 91234'),
+(2, 'Mary Johnson', '[email protected]', '456 Fictional Ave, Springfield, IL 62704'),
+(3, 'John Williams', '[email protected]', '789 Imaginary Ln, Pleasantville, NY 10570'),
+(4, 'Patricia Brown', '[email protected]', '101 Nonexistent Rd, Metropolis, KS 66666'),
+(5, 'Robert Jones', '[email protected]', '222 Make Believe Dr, Smallville, OH 44688'),
+(6, 'Jennifer Garcia', '[email protected]', '333 Phantom Ct, Gotham, NJ 07005'),
+(7, 'Michael Miller', '[email protected]', '444 Unreal Blvd, Wonderland, TX 75001'),
+(8, 'Linda Davis', '[email protected]', '555 Fabricated Way, Neverland, FL 32801'),
+(9, 'William Rodriguez', '[email protected]', '666 Bogus Pl, Oz, KS 67445'),
+(10, 'Elizabeth Martinez', '[email protected]', '777 Sham Ln, Camelot, CA 90210'),
+(11, 'James Johnson', '[email protected]', '888 Pretend Ave, Atlantis, GA 30303'),
+(12, 'Mary Williams', '[email protected]', '999 Simulated Rd, Utopia, MI 48009'),
+(13, 'John Brown', '[email protected]', '1010 Counterfeit St, El Dorado, AR 71730'),
+(14, 'Patricia Jones', '[email protected]', '10 Counterfeit St, El Dorado, AR 71730'),
+(15, 'Robert Garcia', '[email protected]', '1111 Phony Ln, Shangri-La, CO 80014'),
+(16, 'Jennifer Miller', '[email protected]', '1212 Artificial Dr, Rivendell, WA 98101'),
+(17, 'Michael Davis', '[email protected]', '1313 Spurious Ave, Narnia, TN 37201'),
+(18, 'Linda Rodriguez', '[email protected]', '1414 Pseudo Rd, Brigadoon, PA 19003'),
+(19, 'William Martinez', '[email protected]', '1515 Feigned St, Never Never Land, CA 90210'),
+(20, 'Elizabeth Smith', '[email protected]', '1616 Imitation Ln, Asgard, NY 10001'),
+(21, 'James Williams', '[email protected]', '1717 Simulated Ave, Middle Earth, OR 97006'),
+(22, 'Mary Brown', '[email protected]', '123 Fake St, Anytown, CA 91234'),
+(23, 'John Jones', '[email protected]', '456 Fictitious Ave, Springfield, IL 62704'),
+(24, 'Patricia Garcia', '[email protected]', '789 Illusion Ln, Pleasantville, NY 10570'),
+(25, 'Robert Miller', '[email protected]', '101 Imaginary Rd, Metropolis, KS 66666'),
+(26, 'Jennifer Davis', '[email protected]', '222 Make Believe Dr, Neverland, FL 33333'),
+(27, 'Michael Rodriguez', '[email protected]', '333 Pretend Ct, Wonderland, TX 77777'),
+(28, 'Linda Martinez', '[email protected]', '444 Fabricated Blvd, Utopia, WA 98101'),
+(29, 'William Smith', '[email protected]', '555 Sham Way, Mirage, AZ 85001'),
+(30, 'Elizabeth Johnson', '[email protected]', '666 Bogus Pl, Fantasyland, GA 30303'),
+(31, 'James Brown', '[email protected]', '777 Unreal Ave, Dreamville, CO 80202'),
+(32, 'Mary Jones', '[email protected]', '888 Counterfeit Ln, Wishville, OH 44114'),
+(33, 'John Garcia', '[email protected]', '999 Phony Rd, Delusion, MI 48075'),
+(34, 'Patricia Miller', '[email protected]', '1010 Simulated St, Echo, NV 89109'),
+(35, 'Robert Davis', '[email protected]', '1111 Spurious Ave, Replica, PA 19103'),
+(36, 'Jennifer Rodriguez', '[email protected]', '1212 Artificial Dr, Clone, NC 27601'),
+(37, 'Michael Martinez', '[email protected]', '1313 Synthetic Ct, Duplicate, TN 37201'),
+(38, 'Linda Smith', '[email protected]', '1414 Feigned Blvd, Imposter, IN 46204'),
+(39, 'William Johnson', '[email protected]', '1515 Pseudo Pl, Mimic, MN 55401'),
+(40, 'Elizabeth Williams', '[email protected]', '1616 Forged Way, Facsimile, AL 35203')
+) users(id, name, email, address);
+
+
+select * from obfuscate(users, seed=>10) limit 5 offset 20;
+╭────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
+│        id        │        name       │           email           │                 address                 │
+│ Nullable(UInt64) │  Nullable(String) │      Nullable(String)     │             Nullable(String)            │
+├──────────────────┼───────────────────┼───────────────────────────┼─────────────────────────────────────────┤
+│               21 │ William Rodriguez │ michael.davis@example.com │ 1212 Artificial Dr, Rivendell, WA 98101 │
+│               16 │ Jennifer Garcia   │ patricia.brown@gmail      │ 1313 Spurious Ave, NC 27601             │
+│               25 │ John Brown        │ michael.martinez@example  │ 1111 Phony Ln, Asgard, NY 10570         │
+│               30 │ Mary Brown        │ jennifer.garcia@gmail.com │ 222 Make Believe Dr, Clone, NC 27601    │
+│               24 │ James Smith       │ elizabeth.johnson@example │ 444 Fabricated St, Anytown, CA 90210    │
+╰────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
+```

docs/en/sql-reference/20-sql-functions/19-test-functions/sleep.md

@@ -4,9 +4,9 @@ title: SLEEP
 
 Sleeps `seconds` seconds on each data block.
 
-!!! warning 
-    Only used for testing where sleep is required.
-
+:::caution
+Only used for testing where sleep is required.
+:::
 
 ## Syntax
 

docs/en/sql-reference/20-sql-functions/20-other-functions/feistel_obfuscate.md

@@ -0,0 +1,43 @@
+---
+title: FEISTEL_OBFUSCATE
+---
+
+Transformed numbers for anonymization
+
+## Syntax
+
+```sql
+FEISTEL_OBFUSCATE( <number>, <seed> )
+```
+
+## Arguments
+
+| Arguments | Description |
+| ----------- | ----------- |
+| `number` | Input |
+| `seed` | The data for corresponding non-text columns for different tables will be transformed in the same way, so the data for different tables can be JOINed after obfuscation |
+
+## Return Type
+
+Same as input
+
+## Examples
+
+```sql
+SELECT feistel_obfuscate(10000,1561819567875);
++------------------------------------------+
+| feistel_obfuscate(10000, 1561819567875)  |
++------------------------------------------+
+| 15669                                    |
++------------------------------------------+
+```
+
+feistel_obfuscate preserves the number of bits in the original input. If mapping to a larger range is required, an offset can be added to the original input, e.g. feistel_obfuscate(n+10000,50)
+```sql
+SELECT feistel_obfuscate(10,1561819567875);
++------------------------------------------+
+| feistel_obfuscate(10, 1561819567875)     |
++------------------------------------------+
+| 13                                       |
++------------------------------------------+
+```

docs/en/sql-reference/20-sql-functions/20-other-functions/index.md

@@ -14,3 +14,5 @@ This section collects assorted utilities that do not fit into the major function
 | [REMOVE_NULLABLE](remove-nullable.md) | Strip NULLability from a column value |
 | [TO_NULLABLE](to-nullable.md) | Convert a value to a nullable type |
 | [TYPEOF](typeof.md) | Return the name of a value’s data type |
+| [MARKOV_GENERATE](markov_generate.md) | Generate anonymized strings |
+| [FEISTEL_OBFUSCATE](feistel_obfuscate.md) | Transformed numbers for anonymization |

docs/en/sql-reference/20-sql-functions/20-other-functions/markov_generate.md

@@ -0,0 +1,43 @@
+---
+title: MARKOV_GENERATE
+---
+
+Using the model trained by [MARKOV_TRAIN](../07-aggregate-functions/aggregate-markov-train.md) to anonymize the dataset.
+
+## Syntax
+
+```sql
+FEISTEL_OBFUSCATE( <model>, <params>, <seed>, <determinator> )
+```
+
+## Arguments
+
+| Arguments | Description |
+| ----------- | ----------- |
+| `model` | The return model of markov_train |
+| `params`| Json string: `{"order": 5, "sliding_window_size": 8}` <br/> order：order of markov model to generate strings，<br/> size of a sliding window in a source string - its hash is used as a seed for RNG in markov model |
+| `seed` | seed |
+| `determinator`| Source string |
+
+## Return Type
+
+String.
+
+## Examples
+
+```sql
+create table model as
+select markov_train(concat('bar', number::string)) as bar from numbers(100);
+
+select markov_generate(bar,'{"order":5,"sliding_window_size":8}', 151, (number+100000)::string) as generate
+from numbers(5), model;
++-----------+
+| generate  |
++-----------+
+│ bar95     │
+│ bar64     │
+│ bar85     │
+│ bar56     │
+│ bar95     │
++-----------+
+```