README.rst 7.46 KB
Newer Older
Laurent Peuch's avatar
Laurent Peuch committed
1
2
3
4
5
6
7
8
9
10
11
12
13
Cube Doctor
===========

Cube doctor is a tool used to send generated refactoring and QA improvement MR
to logilab's projects.

Usage
-----

CLI pattern looks like:

::

14
    GITLAB_TOKEN=... python3 doctor_hg.py [-h] [-a] [-i] [-m] [-n NUMBER] [-q QUERY] {fix-README,add-tox,add-pytest-deprecated-warnings-cmd,replace-set-attributes-by-set-cw,update-licence-dates}
Laurent Peuch's avatar
Laurent Peuch committed
15
16
17
18
19
20
21
22


Example:

::

    GITLAB_TOKEN=... python3 doctor_hg.py fix-README

Laurent Peuch's avatar
Laurent Peuch committed
23
24
25
26
27
28
29
30
31
32
This GITLAB_TOKEN is an heptapod/gitlab access token. You can manage yours here
https://forge.extranet.logilab.fr/-/profile/personal_access_tokens

Using your token will open MR with your user. If you have enough access this is
the prefered way to do that so we can have an idea on who and how those MR have
been created in case we need to do some debugging.

There is also a cube doctor account with it's own access token, you can ask
people in the team to have it if needed.

Laurent Peuch's avatar
Laurent Peuch committed
33
34
35
36
37
38
39
40
41
Explanation
-----------

The current workflow for cube doctor is:

- select a refactoring command (for example fix-README will rename a README into README.rst and do other needed stuff)
- download our generated heptapod.trig that contains informations about all the public repositories of logilab's forge https://forge.extranet.logilab.fr/
- do a sparql query on that heptapod.trig to select repositories to target (each command has a default query to select only repositories on which it will be able to work)
- alternatively, if no query is provided (but right now all commands provides a query) all cubes of the cubicweb/cube group will be used
42
- for each of those cubes, apply the refactoring command, show what would have been done, but do nothing
Laurent Peuch's avatar
Laurent Peuch committed
43
- except if the "-a"/"--apply" flag is provided then the modification is pushed and a new merge request is created (and it is logged in create_mr.log)
44
- or if the "-i"/"--interactif" flag is provided then for each cube, the user will be prompted to tell if he wants to create a MR
Laurent Peuch's avatar
Laurent Peuch committed
45
- if "-m"/"--merge-when-pipeline-succeeds" is provided along side with "-a" then the merge request will be marked as "merge if pipeline succeeds"
Laurent Peuch's avatar
Laurent Peuch committed
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69

Commands
--------

Current existing refactoring commands

fix-README
~~~~~~~~~~

This command will:

- rename README to README.rst
- replace "README" by "README.rst" in setup.py if present
- replace "README" by "README.rst" in MANIFEST.in if present or add "include README.rst" at the end of it

add-pytest-deprecated-warnings-cmd
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

This command will:

- add :ref:`git+https://github.com/psycojoker/pytest-capture-deprecatedwarnings` to the deps section of the [testenv] of the tox.ini
- or if a dev-requirements.txt exist, add it into it instead
- modify .gitlag-ci.yml "py3" section to add "py3-deprecated-warnings.json" has an artifact

70
71
72
73
74
75
release-new
~~~~~~~~~~~

This command will modify the base tox.ini to add `release-new`.
This tox rule eases the creation of tag version correctly.

Laurent Peuch's avatar
Laurent Peuch committed
76
77
78
79
80
81
82
83
add-tox
~~~~~~~

This command is a bit WIP. It will:

- add a base tox.ini in a repository if it doesn't exist
- it will also launch this tox and fail if the tox fails

84
replace-set-attributes-by-set-cw
Laurent Peuch's avatar
Laurent Peuch committed
85
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
86
87
88
89
90
91

This command will:

- retrieve all the Python files inside the cube
- for each file, replace all occurences of set_attributes calls by set_cw calls

92
update-licence-dates
Laurent Peuch's avatar
Laurent Peuch committed
93
~~~~~~~~~~~~~~~~~~~~
94
95
96
97
98
99

This command will:
- retrieve all the Python files inside the cube
- for each file, if it has a licence part in comments, extend the licence end date to
  the current year.

100
101
102
103
104
105
106
Advanced command
----------------

auto-mr
~~~~~~~

This command will launch all refactoring commands depending of the selected
107
family using "-t"/"--target" (by default "cubes"). **This is meant for the CI**.
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126

The option "-m" can also be used.

run-script
~~~~~~~~~~

This is a Swiss knife command that will allow you either to run:

- a specific bash command using "-k"/"--command"
- or a bash script in a shell using "-s"/"--script" (expecting the file path)
- and commit the result if any modification is made.

On all repositories selected by family.

You NEED to use the "-c"/"--commit", "-b"/"--branch" and either "-s" or "-k"
arguments to use this command.

This is intended for one shot commands (and maybe easier to write commands in the futur).

Laurent Peuch's avatar
Laurent Peuch committed
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
Command CLI arguments
---------------------

All commands share the same CLI arguments. They are:

::

    -a, --apply
        If used a merge request will be created with the modifcations.

        Default value: False

    -i, --interactif
        If used the user will be prompt before created a merge request (a "hg
        diff" and "hg status" will be run before to show the modifications)

        -a/--apply will overwrite this argument

        Default value: False

    -m, --merge-when-pipeline-succeeds
        If used the created merged request will be set to be merged if the
        tests succeeds. Won't have any effect if no merge request is created.

        Default value: False

153
154
155
156
157
158
159
160
161
162
163
164
    -c, --commit
        Allow to overwrite the commit message of the command. Mandatory for the
        "run-script" command.

        Default value: None

    -b, --branch
        Allow to overwrite the branch name of the command. Mandatory for the
        "run-script" command.

        Default value: None

Laurent Peuch's avatar
Laurent Peuch committed
165
166
167
168
169
170
171
172
    -n, --number
        Limit the number of cubes of which to run the command

        Default value: None (equal to "all cubes")

    -q, --query
        Additional conditions to pass to the sparql query SELECT section.

173
174
175
176
177
178
    -p, --project
        If specified, select one specific project instead of using the query.
        Arguments supported are: project id (an it), project url and project name

        Default value: None (unused)

179
180
181
182
183
184
185
    -t, --target
        Can has the value "cubes", "core" or "clients". Will select the family
        of projects specified. Can also be used with "auto-mr" and will skip
        refactoring commands not meant for the specified family

        Default value: cubes

Laurent Peuch's avatar
Laurent Peuch committed
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
Sparql query reference
----------------------

The sparql base query looks like this:

::

    prefix lgg: <http://data.logilab.fr/logigraphe/>
    prefix lon: <http://data.logilab.fr/ontology/network/>
    prefix dep: <http://ontologi.es/doap-deps#>
    prefix doap: <http://usefulinc.com/ns/doap#>

    SELECT ?projectId

    WHERE {
      ?project a doap:Project .
      ?project doap:name ?projectName .
      ?project lgg:heptapod_id ?projectId .
      ?project lgg:is_cube true
    } ORDER BY ?projectName

And all the field of a project looks like this (but this evolve quickly so
check the heptapod.trig to be sure):

::

    lgg:has_black false ;
    lgg:has_capture_deprecated_warnings true ;
    lgg:has_check_manifest true ;
    lgg:has_doc false ;
    lgg:has_flake8 true ;
    lgg:has_from_forge false ;
    lgg:has_gitlab_ci true ;
    lgg:has_mypy false ;
    lgg:has_py2_tests true ;
    lgg:has_py3_tests true ;
    lgg:has_pytest true ;
    lgg:has_readme_rst false ;
    lgg:has_readthedocs false ;
    lgg:has_tox true ;
    lgg:heptapod_id 143 ;
    lgg:is_cube true ;
    lgg:is_new_format true ;
    lgg:last_activity_at "2020-09-17T07:16:55.785000"^^xsd:dateTime ;
    lgg:visibility "public" ;
    dc:description "cubicweb / cubes / activitystream" ;
    dc:title "cubicweb-activitystream" ;
    doap:name "cubicweb-activitystream" ;
    doap:release <https://forge.extranet.logilab.fr/cubicweb/cubes/activitystream/-/releases/0.2.0> .

So, an example query to only select the activitystream could be done like this:

::

    GITLAB_TOKEN=... python3 doctor_hg.py fix-README -q '?project doap:name "cubicweb-activitystream"'