Update [Gemma_3]Activation_Hacking.ipynb

nbfmt

Author: bebechien

Gemma/[Gemma_3]Activation_Hacking.ipynb

@@ -60,28 +60,28 @@
     },
     {
       "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "R4FrgYBWGdsJ"
+      },
+      "outputs": [],
       "source": [
         "#@title Install dependencies\n",
         "! pip install --no-deps -U flax\n",
         "! pip install jaxtyping kagglehub treescope"
-      ],
-      "metadata": {
-        "id": "R4FrgYBWGdsJ"
-      },
-      "execution_count": null,
-      "outputs": []
+      ]
     },
     {
       "cell_type": "markdown",
+      "metadata": {
+        "id": "A0nmk8NwG4fK"
+      },
       "source": [
         "To interact with the Gemma model, you will use the Flax NNX gemma code from\n",
         "google/flax examples on GitHub. Since it is not exposed as a package, you need\n",
         "to use the following workaround to import from the Flax NNX examples/gemma on\n",
         "GitHub."
-      ],
-      "metadata": {
-        "id": "A0nmk8NwG4fK"
-      }
+      ]
     },
     {
       "cell_type": "code",
@@ -121,6 +121,9 @@
     },
     {
       "cell_type": "markdown",
+      "metadata": {
+        "id": "8xUHvwdHH0O1"
+      },
       "source": [
         "To use Gemma model, you’ll need a Kaggle account and API key:\n",
         "\n",
@@ -135,10 +138,7 @@
         "\n",
         "4.  Request access to the model here:\n",
         "    https://www.kaggle.com/models/google/gemma-3"
-      ],
-      "metadata": {
-        "id": "8xUHvwdHH0O1"
-      }
+      ]
     },
     {
       "cell_type": "code",
@@ -275,14 +275,14 @@
     },
     {
       "cell_type": "code",
-      "source": [
-        "prompt_length, out_length, out_data = run_model(\"What is the capital of Switzerland? Answer:\")"
-      ],
+      "execution_count": null,
       "metadata": {
         "id": "i8UVzaiia1pR"
       },
-      "execution_count": null,
-      "outputs": []
+      "outputs": [],
+      "source": [
+        "prompt_length, out_length, out_data = run_model(\"What is the capital of Switzerland? Answer:\")"
+      ]
     },
     {
       "cell_type": "markdown",
@@ -350,8 +350,8 @@
       "cell_type": "code",
       "execution_count": null,
       "metadata": {
-        "id": "N7CMfuIwnR1B",
-        "cellView": "form"
+        "cellView": "form",
+        "id": "N7CMfuIwnR1B"
       },
       "outputs": [],
       "source": [
@@ -435,12 +435,12 @@
     },
     {
       "cell_type": "markdown",
-      "source": [
-        "The next cell will visualize the top activated neurons in a given layer. You can hover over the colored blocks to get the neuron id and value. For the last feedfordward layer _25_ you should see that the top activated neurons are *1937* and *4422*."
-      ],
       "metadata": {
         "id": "c9hYsqK-CTVW"
-      }
+      },
+      "source": [
+        "The next cell will visualize the top activated neurons in a given layer. You can hover over the colored blocks to get the neuron id and value. For the last feedfordward layer _25_ you should see that the top activated neurons are *1937* and *4422*."
+      ]
     },
     {
       "cell_type": "code",
@@ -487,15 +487,14 @@
     },
     {
       "cell_type": "markdown",
+      "metadata": {
+        "id": "30A0KOHTCiVB"
+      },
       "source": [
         "After identifiying a neuron of interest we can deactive or boost it. You can play around with the bias values to achieve different behaviours.\n",
         "1.  **Deactivate** neuron 1937 in layer 25 and issue the same prompt again.\n",
-        "2.  **Boost** neuron 1937 in layer 25 and your model should repsonse with *Switzerland* significantly more often.\n",
-        "\n"
-      ],
-      "metadata": {
-        "id": "30A0KOHTCiVB"
-      }
+        "2.  **Boost** neuron 1937 in layer 25 and your model should repsonse with *Switzerland* significantly more often.\n"
+      ]
     },
     {
       "cell_type": "code",
@@ -531,18 +530,17 @@
         "sampler = sampler_lib.Sampler(\n",
         "    transformer=transformer,\n",
         "    vocab=vocab,\n",
-        ")\n",
-        "\n"
+        ")\n"
       ]
     },
     {
       "cell_type": "markdown",
-      "source": [
-        "We can also apply the `premature_decode` functions to the value of a neuron to investigate the effect of a neuron. Check neuron *1937* of layer *25* to verify why the model behaviour changed by deactivating or boosting the neuron."
-      ],
       "metadata": {
         "id": "CNWJ-8YzDCxw"
-      }
+      },
+      "source": [
+        "We can also apply the `premature_decode` functions to the value of a neuron to investigate the effect of a neuron. Check neuron *1937* of layer *25* to verify why the model behaviour changed by deactivating or boosting the neuron."
+      ]
     },
     {
       "cell_type": "code",
@@ -585,8 +583,8 @@
       "cell_type": "code",
       "execution_count": null,
       "metadata": {
-        "id": "2Cjc8YMM-fov",
-        "cellView": "form"
+        "cellView": "form",
+        "id": "2Cjc8YMM-fov"
       },
       "outputs": [],
       "source": [
@@ -699,20 +697,15 @@
     }
   ],
   "metadata": {
+    "accelerator": "GPU",
     "colab": {
-      "private_outputs": true,
-      "provenance": [],
-      "gpuType": "A100",
-      "machine_shape": "hm"
+      "name": "[Gemma_3]Activation_Hacking.ipynb",
+      "toc_visible": true
     },
     "kernelspec": {
       "display_name": "Python 3",
       "name": "python3"
-    },
-    "language_info": {
-      "name": "python"
-    },
-    "accelerator": "GPU"
+    }
   },
   "nbformat": 4,
   "nbformat_minor": 0