README.rst 6.99 KB
Newer Older
Laurent Peuch's avatar
Laurent Peuch committed
1
2
3
4
5
6
7
8
9
10
11
12
13
Cube Doctor
===========

Cube doctor is a tool used to send generated refactoring and QA improvement MR
to logilab's projects.

Usage
-----

CLI pattern looks like:

::

14
    GITLAB_TOKEN=... python3 doctor_hg.py [-h] [-a] [-i] [-m] [-n NUMBER] [-q QUERY] {fix-README,add-tox,add-pytest-deprecated-warnings-cmd,replace-set-attributes-by-set-cw,update-licence-dates}
Laurent Peuch's avatar
Laurent Peuch committed
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31


Example:

::

    GITLAB_TOKEN=... python3 doctor_hg.py fix-README

Explanation
-----------

The current workflow for cube doctor is:

- select a refactoring command (for example fix-README will rename a README into README.rst and do other needed stuff)
- download our generated heptapod.trig that contains informations about all the public repositories of logilab's forge https://forge.extranet.logilab.fr/
- do a sparql query on that heptapod.trig to select repositories to target (each command has a default query to select only repositories on which it will be able to work)
- alternatively, if no query is provided (but right now all commands provides a query) all cubes of the cubicweb/cube group will be used
32
- for each of those cubes, apply the refactoring command, show what would have been done, but do nothing
Laurent Peuch's avatar
Laurent Peuch committed
33
- except if the "-a"/"--apply" flag is provided then the modification is pushed and a new merge request is created (and it is logged in create_mr.log)
34
- or if the "-i"/"--interactif" flag is provided then for each cube, the user will be prompted to tell if he wants to create a MR
Laurent Peuch's avatar
Laurent Peuch committed
35
- if "-m"/"--merge-when-pipeline-succeeds" is provided along side with "-a" then the merge request will be marked as "merge if pipeline succeeds"
Laurent Peuch's avatar
Laurent Peuch committed
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59

Commands
--------

Current existing refactoring commands

fix-README
~~~~~~~~~~

This command will:

- rename README to README.rst
- replace "README" by "README.rst" in setup.py if present
- replace "README" by "README.rst" in MANIFEST.in if present or add "include README.rst" at the end of it

add-pytest-deprecated-warnings-cmd
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

This command will:

- add :ref:`git+https://github.com/psycojoker/pytest-capture-deprecatedwarnings` to the deps section of the [testenv] of the tox.ini
- or if a dev-requirements.txt exist, add it into it instead
- modify .gitlag-ci.yml "py3" section to add "py3-deprecated-warnings.json" has an artifact

60
61
62
63
64
65
release-new
~~~~~~~~~~~

This command will modify the base tox.ini to add `release-new`.
This tox rule eases the creation of tag version correctly.

Laurent Peuch's avatar
Laurent Peuch committed
66
67
68
69
70
71
72
73
add-tox
~~~~~~~

This command is a bit WIP. It will:

- add a base tox.ini in a repository if it doesn't exist
- it will also launch this tox and fail if the tox fails

74
replace-set-attributes-by-set-cw
Laurent Peuch's avatar
Laurent Peuch committed
75
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
76
77
78
79
80
81

This command will:

- retrieve all the Python files inside the cube
- for each file, replace all occurences of set_attributes calls by set_cw calls

82
update-licence-dates
Laurent Peuch's avatar
Laurent Peuch committed
83
~~~~~~~~~~~~~~~~~~~~
84
85
86
87
88
89

This command will:
- retrieve all the Python files inside the cube
- for each file, if it has a licence part in comments, extend the licence end date to
  the current year.

90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
Advanced command
----------------

auto-mr
~~~~~~~

This command will launch all refactoring commands depending of the selected
family using "-t"/"--target" (by default "cubes"). This is meant for the ci.

The option "-m" can also be used.

run-script
~~~~~~~~~~

This is a Swiss knife command that will allow you either to run:

- a specific bash command using "-k"/"--command"
- or a bash script in a shell using "-s"/"--script" (expecting the file path)
- and commit the result if any modification is made.

On all repositories selected by family.

You NEED to use the "-c"/"--commit", "-b"/"--branch" and either "-s" or "-k"
arguments to use this command.

This is intended for one shot commands (and maybe easier to write commands in the futur).

Laurent Peuch's avatar
Laurent Peuch committed
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
Command CLI arguments
---------------------

All commands share the same CLI arguments. They are:

::

    -a, --apply
        If used a merge request will be created with the modifcations.

        Default value: False

    -i, --interactif
        If used the user will be prompt before created a merge request (a "hg
        diff" and "hg status" will be run before to show the modifications)

        -a/--apply will overwrite this argument

        Default value: False

    -m, --merge-when-pipeline-succeeds
        If used the created merged request will be set to be merged if the
        tests succeeds. Won't have any effect if no merge request is created.

        Default value: False

143
144
145
146
147
148
149
150
151
152
153
154
    -c, --commit
        Allow to overwrite the commit message of the command. Mandatory for the
        "run-script" command.

        Default value: None

    -b, --branch
        Allow to overwrite the branch name of the command. Mandatory for the
        "run-script" command.

        Default value: None

Laurent Peuch's avatar
Laurent Peuch committed
155
156
157
158
159
160
161
162
    -n, --number
        Limit the number of cubes of which to run the command

        Default value: None (equal to "all cubes")

    -q, --query
        Additional conditions to pass to the sparql query SELECT section.

163
164
165
166
167
168
    -p, --project
        If specified, select one specific project instead of using the query.
        Arguments supported are: project id (an it), project url and project name

        Default value: None (unused)

169
170
171
172
173
174
175
    -t, --target
        Can has the value "cubes", "core" or "clients". Will select the family
        of projects specified. Can also be used with "auto-mr" and will skip
        refactoring commands not meant for the specified family

        Default value: cubes

Laurent Peuch's avatar
Laurent Peuch committed
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
Sparql query reference
----------------------

The sparql base query looks like this:

::

    prefix lgg: <http://data.logilab.fr/logigraphe/>
    prefix lon: <http://data.logilab.fr/ontology/network/>
    prefix dep: <http://ontologi.es/doap-deps#>
    prefix doap: <http://usefulinc.com/ns/doap#>

    SELECT ?projectId

    WHERE {
      ?project a doap:Project .
      ?project doap:name ?projectName .
      ?project lgg:heptapod_id ?projectId .
      ?project lgg:is_cube true
    } ORDER BY ?projectName

And all the field of a project looks like this (but this evolve quickly so
check the heptapod.trig to be sure):

::

    lgg:has_black false ;
    lgg:has_capture_deprecated_warnings true ;
    lgg:has_check_manifest true ;
    lgg:has_doc false ;
    lgg:has_flake8 true ;
    lgg:has_from_forge false ;
    lgg:has_gitlab_ci true ;
    lgg:has_mypy false ;
    lgg:has_py2_tests true ;
    lgg:has_py3_tests true ;
    lgg:has_pytest true ;
    lgg:has_readme_rst false ;
    lgg:has_readthedocs false ;
    lgg:has_tox true ;
    lgg:heptapod_id 143 ;
    lgg:is_cube true ;
    lgg:is_new_format true ;
    lgg:last_activity_at "2020-09-17T07:16:55.785000"^^xsd:dateTime ;
    lgg:visibility "public" ;
    dc:description "cubicweb / cubes / activitystream" ;
    dc:title "cubicweb-activitystream" ;
    doap:name "cubicweb-activitystream" ;
    doap:release <https://forge.extranet.logilab.fr/cubicweb/cubes/activitystream/-/releases/0.2.0> .

So, an example query to only select the activitystream could be done like this:

::

    GITLAB_TOKEN=... python3 doctor_hg.py fix-README -q '?project doap:name "cubicweb-activitystream"'