Commit 0b556b5c authored by Laurent Peuch's avatar Laurent Peuch
Browse files

Update README.rst

parent 033049ebacf7
Pipeline #41900 passed with stage
in 1 minute
Cube Doctor
===========
Cube doctor is a tool used to send generated refactoring and QA improvement MR
to logilab's projects.
Usage
-----
CLI pattern looks like:
::
GITLAB_TOKEN=... python3 doctor_hg.py [-h] [-a] [-i] [-m] [-n NUMBER] [-q QUERY] {fix-README,add-tox,add-pytest-deprecated-warnings-cmd,replace-set-attributes-by-set-cw,update-licence-dates}
Example:
::
GITLAB_TOKEN=... python3 doctor_hg.py fix-README
Explanation
-----------
The current workflow for cube doctor is:
- select a refactoring command (for example fix-README will rename a README into README.rst and do other needed stuff)
- download our generated heptapod.trig that contains informations about all the public repositories of logilab's forge https://forge.extranet.logilab.fr/
- do a sparql query on that heptapod.trig to select repositories to target (each command has a default query to select only repositories on which it will be able to work)
- alternatively, if no query is provided (but right now all commands provides a query) all cubes of the cubicweb/cube group will be used
- for each of those cubes, apply the refactoring command, show what would have been done, but do nothing
- except if the "-a"/"--apply" flag is provided then the modification is pushed and a new merge request is created (and it is logged in create_mr.log)
- or if the "-i"/"--interactif" flag is provided then for each cube, the user will be prompted to tell if he wants to create a MR
- if "-m"/"--merge-when-pipeline-succeeds" is provided along side with "-a" then the merge request will be marked as "merge if pipeline succeeds"
Commands
--------
Current existing refactoring commands
fix-README
~~~~~~~~~~
This command will:
- rename README to README.rst
- replace "README" by "README.rst" in setup.py if present
- replace "README" by "README.rst" in MANIFEST.in if present or add "include README.rst" at the end of it
add-pytest-deprecated-warnings-cmd
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
This command will:
- add :ref:`git+https://github.com/psycojoker/pytest-capture-deprecatedwarnings` to the deps section of the [testenv] of the tox.ini
- or if a dev-requirements.txt exist, add it into it instead
- modify .gitlag-ci.yml "py3" section to add "py3-deprecated-warnings.json" has an artifact
add-tox
~~~~~~~
This command is a bit WIP. It will:
- add a base tox.ini in a repository if it doesn't exist
- it will also launch this tox and fail if the tox fails
replace-set-attributes-by-set-cw
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
This command will:
- retrieve all the Python files inside the cube
- for each file, replace all occurences of set_attributes calls by set_cw calls
update-licence-dates
~~~~~~~~~~~~~~~~~~~~
This command will:
- retrieve all the Python files inside the cube
- for each file, if it has a licence part in comments, extend the licence end date to
the current year.
Advanced command
----------------
auto-mr
~~~~~~~
This command will launch all refactoring commands depending of the selected
family using "-t"/"--target" (by default "cubes"). This is meant for the ci.
The option "-m" can also be used.
run-script
~~~~~~~~~~
This is a Swiss knife command that will allow you either to run:
- a specific bash command using "-k"/"--command"
- or a bash script in a shell using "-s"/"--script" (expecting the file path)
- and commit the result if any modification is made.
On all repositories selected by family.
You NEED to use the "-c"/"--commit", "-b"/"--branch" and either "-s" or "-k"
arguments to use this command.
This is intended for one shot commands (and maybe easier to write commands in the futur).
Command CLI arguments
---------------------
All commands share the same CLI arguments. They are:
::
-a, --apply
If used a merge request will be created with the modifcations.
Default value: False
-i, --interactif
If used the user will be prompt before created a merge request (a "hg
diff" and "hg status" will be run before to show the modifications)
-a/--apply will overwrite this argument
Default value: False
-m, --merge-when-pipeline-succeeds
If used the created merged request will be set to be merged if the
tests succeeds. Won't have any effect if no merge request is created.
Default value: False
-c, --commit
Allow to overwrite the commit message of the command. Mandatory for the
"run-script" command.
Default value: None
-b, --branch
Allow to overwrite the branch name of the command. Mandatory for the
"run-script" command.
Default value: None
-n, --number
Limit the number of cubes of which to run the command
Default value: None (equal to "all cubes")
-q, --query
Additional conditions to pass to the sparql query SELECT section.
-p, --project
If specified, select one specific project instead of using the query.
Arguments supported are: project id (an it), project url and project name
Default value: None (unused)
-t, --target
Can has the value "cubes", "core" or "clients". Will select the family
of projects specified. Can also be used with "auto-mr" and will skip
refactoring commands not meant for the specified family
Default value: cubes
Sparql query reference
----------------------
The sparql base query looks like this:
::
prefix lgg: <http://data.logilab.fr/logigraphe/>
prefix lon: <http://data.logilab.fr/ontology/network/>
prefix dep: <http://ontologi.es/doap-deps#>
prefix doap: <http://usefulinc.com/ns/doap#>
SELECT ?projectId
WHERE {
?project a doap:Project .
?project doap:name ?projectName .
?project lgg:heptapod_id ?projectId .
?project lgg:is_cube true
} ORDER BY ?projectName
And all the field of a project looks like this (but this evolve quickly so
check the heptapod.trig to be sure):
::
lgg:has_black false ;
lgg:has_capture_deprecated_warnings true ;
lgg:has_check_manifest true ;
lgg:has_doc false ;
lgg:has_flake8 true ;
lgg:has_from_forge false ;
lgg:has_gitlab_ci true ;
lgg:has_mypy false ;
lgg:has_py2_tests true ;
lgg:has_py3_tests true ;
lgg:has_pytest true ;
lgg:has_readme_rst false ;
lgg:has_readthedocs false ;
lgg:has_tox true ;
lgg:heptapod_id 143 ;
lgg:is_cube true ;
lgg:is_new_format true ;
lgg:last_activity_at "2020-09-17T07:16:55.785000"^^xsd:dateTime ;
lgg:visibility "public" ;
dc:description "cubicweb / cubes / activitystream" ;
dc:title "cubicweb-activitystream" ;
doap:name "cubicweb-activitystream" ;
doap:release <https://forge.extranet.logilab.fr/cubicweb/cubes/activitystream/-/releases/0.2.0> .
So, an example query to only select the activitystream could be done like this:
::
GITLAB_TOKEN=... python3 doctor_hg.py fix-README -q '?project doap:name "cubicweb-activitystream"'
Cube Doctor
===========
Cube doctor is a tool used to send generated refactoring and QA improvement MR
to logilab's projects.
Usage
-----
CLI pattern looks like:
::
GITLAB_TOKEN=... python3 doctor_hg.py [-h] [-a] [-i] [-m] [-n NUMBER] [-q QUERY] {fix-README,add-tox,add-pytest-deprecated-warnings-cmd,replace-set-attributes-by-set-cw,update-licence-dates}
Example:
::
GITLAB_TOKEN=... python3 doctor_hg.py fix-README
Explanation
-----------
The current workflow for cube doctor is:
- select a refactoring command (for example fix-README will rename a README into README.rst and do other needed stuff)
- download our generated heptapod.trig that contains informations about all the public repositories of logilab's forge https://forge.extranet.logilab.fr/
- do a sparql query on that heptapod.trig to select repositories to target (each command has a default query to select only repositories on which it will be able to work)
- alternatively, if no query is provided (but right now all commands provides a query) all cubes of the cubicweb/cube group will be used
- for each of those cubes, apply the refactoring command, show what would have been done, but do nothing
- except if the "-a"/"--apply" flag is provided then the modification is pushed and a new merge request is created (and it is logged in create_mr.log)
- or if the "-i"/"--interactif" flag is provided then for each cube, the user will be prompted to tell if he wants to create a MR
- if "-m"/"--merge-when-pipeline-succeeds" is provided along side with "-a" then the merge request will be marked as "merge if pipeline succeeds"
Commands
--------
Current existing refactoring commands
fix-README
~~~~~~~~~~
This command will:
- rename README to README.rst
- replace "README" by "README.rst" in setup.py if present
- replace "README" by "README.rst" in MANIFEST.in if present or add "include README.rst" at the end of it
add-pytest-deprecated-warnings-cmd
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
This command will:
- add :ref:`git+https://github.com/psycojoker/pytest-capture-deprecatedwarnings` to the deps section of the [testenv] of the tox.ini
- or if a dev-requirements.txt exist, add it into it instead
- modify .gitlag-ci.yml "py3" section to add "py3-deprecated-warnings.json" has an artifact
add-tox
~~~~~~~
This command is a bit WIP. It will:
- add a base tox.ini in a repository if it doesn't exist
- it will also launch this tox and fail if the tox fails
replace-set-attributes-by-set-cw
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
This command will:
- retrieve all the Python files inside the cube
- for each file, replace all occurences of set_attributes calls by set_cw calls
update-licence-dates
~~~~~~~~~~~~~~~~~~~~
This command will:
- retrieve all the Python files inside the cube
- for each file, if it has a licence part in comments, extend the licence end date to
the current year.
Advanced command
----------------
auto-mr
~~~~~~~
This command will launch all refactoring commands depending of the selected
family using "-t"/"--target" (by default "cubes"). This is meant for the ci.
The option "-m" can also be used.
run-script
~~~~~~~~~~
This is a Swiss knife command that will allow you either to run:
- a specific bash command using "-k"/"--command"
- or a bash script in a shell using "-s"/"--script" (expecting the file path)
- and commit the result if any modification is made.
On all repositories selected by family.
You NEED to use the "-c"/"--commit", "-b"/"--branch" and either "-s" or "-k"
arguments to use this command.
This is intended for one shot commands (and maybe easier to write commands in the futur).
Command CLI arguments
---------------------
All commands share the same CLI arguments. They are:
::
-a, --apply
If used a merge request will be created with the modifcations.
Default value: False
-i, --interactif
If used the user will be prompt before created a merge request (a "hg
diff" and "hg status" will be run before to show the modifications)
-a/--apply will overwrite this argument
Default value: False
-m, --merge-when-pipeline-succeeds
If used the created merged request will be set to be merged if the
tests succeeds. Won't have any effect if no merge request is created.
Default value: False
-c, --commit
Allow to overwrite the commit message of the command. Mandatory for the
"run-script" command.
Default value: None
-b, --branch
Allow to overwrite the branch name of the command. Mandatory for the
"run-script" command.
Default value: None
-n, --number
Limit the number of cubes of which to run the command
Default value: None (equal to "all cubes")
-q, --query
Additional conditions to pass to the sparql query SELECT section.
-p, --project
If specified, select one specific project instead of using the query.
Arguments supported are: project id (an it), project url and project name
Default value: None (unused)
-t, --target
Can has the value "cubes", "core" or "clients". Will select the family
of projects specified. Can also be used with "auto-mr" and will skip
refactoring commands not meant for the specified family
Default value: cubes
Sparql query reference
----------------------
The sparql base query looks like this:
::
prefix lgg: <http://data.logilab.fr/logigraphe/>
prefix lon: <http://data.logilab.fr/ontology/network/>
prefix dep: <http://ontologi.es/doap-deps#>
prefix doap: <http://usefulinc.com/ns/doap#>
SELECT ?projectId
WHERE {
?project a doap:Project .
?project doap:name ?projectName .
?project lgg:heptapod_id ?projectId .
?project lgg:is_cube true
} ORDER BY ?projectName
And all the field of a project looks like this (but this evolve quickly so
check the heptapod.trig to be sure):
::
lgg:has_black false ;
lgg:has_capture_deprecated_warnings true ;
lgg:has_check_manifest true ;
lgg:has_doc false ;
lgg:has_flake8 true ;
lgg:has_from_forge false ;
lgg:has_gitlab_ci true ;
lgg:has_mypy false ;
lgg:has_py2_tests true ;
lgg:has_py3_tests true ;
lgg:has_pytest true ;
lgg:has_readme_rst false ;
lgg:has_readthedocs false ;
lgg:has_tox true ;
lgg:heptapod_id 143 ;
lgg:is_cube true ;
lgg:is_new_format true ;
lgg:last_activity_at "2020-09-17T07:16:55.785000"^^xsd:dateTime ;
lgg:visibility "public" ;
dc:description "cubicweb / cubes / activitystream" ;
dc:title "cubicweb-activitystream" ;
doap:name "cubicweb-activitystream" ;
doap:release <https://forge.extranet.logilab.fr/cubicweb/cubes/activitystream/-/releases/0.2.0> .
So, an example query to only select the activitystream could be done like this:
::
GITLAB_TOKEN=... python3 doctor_hg.py fix-README -q '?project doap:name "cubicweb-activitystream"'
test endl
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment