Allow uninstalling Jenkins plugins

[fredg02]

Allow uninstalling Jenkins plugins and handle associated permissions (e.g. Gerrit) to avoid JCasC startup errors.

This might become easier with the new Jenkins Plugins Installation Manager CLI Tool/Library: https://jenkins.io/projects/gsoc/2019/plugin-installation-manager-tool-cli/

[mbarbero]

Currently, here is how plugins are installed/managed between the container images and the running instances:

• A jiro-master image contains some plugins as listed in https://github.com/eclipse-cbi/jiro-masters/blob/master/jiro.libsonnet#L45. Plugins are downloaded (along with their dependencies) via a shell script (soon via the plugin manager cli, see Switch to plugin-installation-manager-tool for downloading plugins jiro-masters#3) in /usr/share/jenkins/ref/plugins.
• a specific instance image take the jiro-master image and add a layer with additional plugins (and deps) from its plugins-list file (if any, e.g. https://github.com/eclipse-cbi/jiro/blob/master/instances/technology.cbi/jenkins/plugins-list) and put them in /usr/share/jenkins/ref/plugins as well. Thus, an instance image has all its plugins in /usr/share/jenkins/ref/plugins.
• When the instance starts, the jenkins.sh(https://github.com/eclipse-cbi/jiro-masters/blob/master/jiro.libsonnet#L40) is called (https://github.com/eclipse-cbi/jiro-masters/blob/master/Dockerfile#L24) that will first reconcile what's in the image's /usr/share/jenkins/ref/plugins folder and what's in the actual $JENKINS_HOME/plugins (which is a persistent volume).

In order to be able to remove a plugin, we need to be able to remove it from $JENKINS_HOME/plugins folder in the persistent volume. I see 3 options:

1. We add a directive to the instance's config.jsonnet with the list of plugins to be removed (this list could be copied to some specific file in the /usr/share/jenkins/ref/ folder) and we add another responsibility to jenkins.sh to actually remove the plugins from this file. I'm not a huge fan as the list of plugins to be removed is not something "stable". There is no reason to keep a plugin in this list once it has been removed... It's also quite complicated as we would have to check if the plugins we want to remove are not a dependencies for some other plugins.
2. We removed all plugins from $JENKINS_HOME/plugins at each startup, so that the re-conciliation between /usr/share/jenkins/ref/plugins and $JENKINS_HOME/plugins is fairly simple: it's just a copy. Of course, it means that we don't support persistent installation of plugins via the UI. We can still test new plugin and install them from the UI, but once the instance is restarted, the plugin is gone if not added to the image. I like that as it's very close to how we handle configuration via JCasC: all manual edits are overridden at restart. Finally, it's much safer as we don't risk removing a plugin that would end up being a dependency of another.
3. We do it manually: we first remove the desired plugin either from the jiro-master's list or from the instance's list, then we re-deploy the instance. Once restarted, we go to the plugin management UI and remove the plugin from there and re-restart the instance again. The plugin should be gone. It's manageable for the removal of a plugin in 1 or 2 instances, but it does not scale very well (e.g. removal of a deprecated plugin from all instances).

@fredg02, WDYT?

[heurtematte]

I agree on second proposition.

[fredg02]

I'd like to propose another option:

4. Manual approach, assisted by scripts to allow (mass) plugin uninstallations
   We don't run the Jenkins instances completely stateless, since JCasC is not perfect yet. I'd also rather have the option to install/update a plugin from the UI manually, then an automatic reset/override after a restart. We have (Groovy) scripts that can mass uninstall plugins if required. Those scripts could also be leveraged to uninstall plugins on a single instance if someone does not like to click in the WebUI. In any case, this will need to be documented better.

[heurtematte]

Maybe we should learn from this issue: https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/issues/3650
Keeping an old version of "maven-plugin" on the jenkins home plugins directory failed jenkins to start.
Cleaning directory plugins was the workaround.