Cube Doctor
===========
Cube doctor is a tool used to send generated refactoring and QA improvement MR
to logilab's projects.
Usage
-----
CLI pattern looks like:
::
GITLAB_TOKEN=... python3 doctor_hg.py [-h] [-a] [-i] [-m] [-n NUMBER] [-q QUERY] {fix-README,add-tox,add-pytest-deprecated-warnings-cmd,replace-set-attributes-by-set-cw,update-licence-dates}
Example:
::
GITLAB_TOKEN=... python3 doctor_hg.py fix-README
This GITLAB_TOKEN is an heptapod/gitlab access token. You can manage yours here
https://forge.extranet.logilab.fr/-/profile/personal_access_tokens
Using your token will open MR with your user. If you have enough access this is
the prefered way to do that so we can have an idea on who and how those MR have
been created in case we need to do some debugging.
There is also a cube doctor account with it's own access token, you can ask
people in the team to have it if needed.
Explanation
-----------
The current workflow for cube doctor is:
- select a refactoring command (for example fix-README will rename a README into README.rst and do other needed stuff)
- download our generated heptapod.trig that contains informations about all the public repositories of logilab's forge https://forge.extranet.logilab.fr/
- do a sparql query on that heptapod.trig to select repositories to target (each command has a default query to select only repositories on which it will be able to work)
- alternatively, if no query is provided (but right now all commands provides a query) all cubes of the cubicweb/cube group will be used
- for each of those cubes, apply the refactoring command, show what would have been done, but do nothing
- except if the "-a"/"--apply" flag is provided then the modification is pushed and a new merge request is created (and it is logged in create_mr.log)
- or if the "-i"/"--interactif" flag is provided then for each cube, the user will be prompted to tell if he wants to create a MR
- if "-m"/"--merge-when-pipeline-succeeds" is provided along side with "-a" then the merge request will be marked as "merge if pipeline succeeds"
Commands
--------
Current existing refactoring commands
fix-README
~~~~~~~~~~
This command will:
- rename README to README.rst
- replace "README" by "README.rst" in setup.py if present
- replace "README" by "README.rst" in MANIFEST.in if present or add "include README.rst" at the end of it
add-pytest-deprecated-warnings-cmd
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
This command will:
- add :ref:`git+https://github.com/psycojoker/pytest-capture-deprecatedwarnings` to the deps section of the [testenv] of the tox.ini
- or if a dev-requirements.txt exist, add it into it instead
- modify .gitlag-ci.yml "py3" section to add "py3-deprecated-warnings.json" has an artifact
release-new
~~~~~~~~~~~
This command will modify the base tox.ini to add `release-new`.
This tox rule eases the creation of tag version correctly.
add-tox
~~~~~~~
This command is a bit WIP. It will:
- add a base tox.ini in a repository if it doesn't exist
- it will also launch this tox and fail if the tox fails
replace-set-attributes-by-set-cw
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
This command will:
- retrieve all the Python files inside the cube
- for each file, replace all occurences of set_attributes calls by set_cw calls
update-licence-dates
~~~~~~~~~~~~~~~~~~~~
This command will:
- retrieve all the Python files inside the cube
- for each file, if it has a licence part in comments, extend the licence end date to
the current year.
Advanced command
----------------
auto-mr
~~~~~~~
This command will launch all refactoring commands depending of the selected
family using "-t"/"--target" (by default "cubes"). **This is meant for the CI**.
The option "-m" can also be used.
run-script
~~~~~~~~~~
This is a Swiss knife command that will allow you either to run:
- a specific bash command using "-k"/"--command"
- or a bash script in a shell using "-s"/"--script" (expecting the file path)
- and commit the result if any modification is made.
On all repositories selected by family.
You NEED to use the "-c"/"--commit", "-b"/"--branch" and either "-s" or "-k"
arguments to use this command.
This is intended for one shot commands (and maybe easier to write commands in the futur).
Command CLI arguments
---------------------
All commands share the same CLI arguments. They are:
::
-a, --apply
If used a merge request will be created with the modifcations.
Default value: False
-i, --interactif
If used the user will be prompt before created a merge request (a "hg
diff" and "hg status" will be run before to show the modifications)
-a/--apply will overwrite this argument
Default value: False
-m, --merge-when-pipeline-succeeds
If used the created merged request will be set to be merged if the
tests succeeds. Won't have any effect if no merge request is created.
Default value: False
-c, --commit
Allow to overwrite the commit message of the command. Mandatory for the
"run-script" command.
Default value: None
-b, --branch
Allow to overwrite the branch name of the command. Mandatory for the
"run-script" command.
Default value: None
-n, --number
Limit the number of cubes of which to run the command
Default value: None (equal to "all cubes")
-q, --query
Additional conditions to pass to the sparql query SELECT section.
-p, --project
If specified, select one specific project instead of using the query.
Arguments supported are: project id (an it), project url and project name
Default value: None (unused)
-t, --target
Can has the value "cubes", "core" or "clients". Will select the family
of projects specified. Can also be used with "auto-mr" and will skip
refactoring commands not meant for the specified family
Default value: cubes
Sparql query reference
----------------------
The sparql base query looks like this:
::
prefix lgg:
prefix lon:
prefix dep:
prefix doap:
SELECT ?projectId
WHERE {
?project a doap:Project .
?project doap:name ?projectName .
?project lgg:heptapod_id ?projectId .
?project lgg:is_cube true
} ORDER BY ?projectName
And all the field of a project looks like this (but this evolve quickly so
check the heptapod.trig to be sure):
::
lgg:has_black false ;
lgg:has_capture_deprecated_warnings true ;
lgg:has_check_manifest true ;
lgg:has_doc false ;
lgg:has_flake8 true ;
lgg:has_from_forge false ;
lgg:has_gitlab_ci true ;
lgg:has_mypy false ;
lgg:has_py2_tests true ;
lgg:has_py3_tests true ;
lgg:has_pytest true ;
lgg:has_readme_rst false ;
lgg:has_readthedocs false ;
lgg:has_tox true ;
lgg:heptapod_id 143 ;
lgg:is_cube true ;
lgg:is_new_format true ;
lgg:last_activity_at "2020-09-17T07:16:55.785000"^^xsd:dateTime ;
lgg:visibility "public" ;
dc:description "cubicweb / cubes / activitystream" ;
dc:title "cubicweb-activitystream" ;
doap:name "cubicweb-activitystream" ;
doap:release .
So, an example query to only select the activitystream could be done like this:
::
GITLAB_TOKEN=... python3 doctor_hg.py fix-README -q '?project doap:name "cubicweb-activitystream"'