Introduction
This document describes how to add regular expression search support to your roundup tracker. The syntax for the regular expressions is of course the same syntax as used in Python's RE module. For more information about the syntax, we refer to the documentation belonging to that module.
Requirements
The DatabaseWrapper is required if you want to add the regular expression search to your tracker.
Implementation
The regular expression feature consists out of a new search HTML template and a new action handler that supports regular expressions.
Search HTML Template
The new template is based on the 'issue.search.html' template and it came out of the roundup 1.1.1 distribution. To see the changes, please run a diff against the new 'issue.search.html' template and the out of the box one from 1.1.1. Don't get scared. The changes are simple and not too much.
Action Handler
The action handler is also based on a standard out of the box action handler of roundup 1.1.1. That action handler is 'SearchAction' in 'actions.py'. The new handler supports all the features of the original handler as long as the user didn't ask for a regular expression search. To see the changes, please run a diff against the original action handler.
Best regards,
Marlon van den Berg
PS: Be aware that a regular expression search can consume some time when you run it on a slow server or in a tracker with a huge number of issues and messages.
Source Code
Here is the new 'issue.search.html' template:
1 <tal:block metal:use-macro="templates/page/macros/icing">
2 <title metal:fill-slot="head_title" i18n:translate="">Issue searching - <span
3 i18n:name="tracker" tal:replace="config/TRACKER_NAME" /></title>
4 <span metal:fill-slot="body_title" tal:omit-tag="python:1"
5 i18n:translate="">Issue searching</span>
6 <td class="content" metal:fill-slot="content">
7
8 <disabled script language="javascript" type="text/javascript">
9 //<!--
10 function onClick_RegExp() {
11 if (self.document.itemSynopsis['reg_exp'].checked) {
12 self.document.itemSynopsis['re_ignorecase'].disabled = false;
13 } else {
14 self.document.itemSynopsis['re_ignorecase'].checked = false;
15 self.document.itemSynopsis['re_ignorecase'].disabled = true;
16 }
17 }
18 //-->
19 <disabled /script>
20
21 <form method="GET" name="itemSynopsis"
22 tal:attributes="action request/classname">
23
24 <table class="form" tal:define="
25 cols python:request.columns or 'id activity title status assignedto'.split();
26 sort_on python:request.sort[1] or 'activity';
27 group_on python:request.group[1] or 'priority';
28
29 search_input templates/page/macros/search_input;
30 column_input templates/page/macros/column_input;
31 sort_input templates/page/macros/sort_input;
32 group_input templates/page/macros/group_input;
33 search_select templates/page/macros/search_select;
34 search_multiselect templates/page/macros/search_multiselect;">
35
36 <tr tal:define="name string:@search_text">
37 <th i18n:translate="">All text*:</th>
38 <td>
39 <input size="45"
40 tal:attributes="value python:request.form.getvalue(name) or nothing;
41 name name">
42 </td>
43 <td nowrap colspan="3">
44 <input type="checkbox" name="reg_exp" value="1" onClick="javascript: onClick_RegExp()"
45 tal:attributes="checked reguest/form/reg_exp/value | nothing">
46 Regular Expression<br>
47 <input type="checkbox" name="re_ignorecase" value="1"
48 tal:attributes="checked reguest/form/re_ignorecase/value | nothing">
49 Ignore case<br>
50 <br>
51 </td>
52 </tr>
53
54 <tr>
55 <th class="header"> </th>
56 <th class="header" i18n:translate="">Filter on</th>
57 <th class="header" i18n:translate="">Display</th>
58 <th class="header" i18n:translate="">Sort on</th>
59 <th class="header" i18n:translate="">Group on</th>
60 </tr>
61
62 <tr tal:define="name string:title">
63 <th i18n:translate="">Title:</th>
64 <td metal:use-macro="search_input"></td>
65 <td metal:use-macro="column_input"></td>
66 <td metal:use-macro="sort_input"></td>
67 <td> </td>
68 </tr>
69
70 <tr tal:define="name string:topic;
71 db_klass string:keyword;
72 db_content string:name;">
73 <th i18n:translate="">Topic:</th>
74 <td metal:use-macro="search_select"></td>
75 <td metal:use-macro="column_input"></td>
76 <td metal:use-macro="sort_input"></td>
77 <td metal:use-macro="group_input"></td>
78 </tr>
79
80 <tr tal:define="name string:id">
81 <th i18n:translate="">ID:</th>
82 <td metal:use-macro="search_input"></td>
83 <td metal:use-macro="column_input"></td>
84 <td metal:use-macro="sort_input"></td>
85 <td> </td>
86 </tr>
87
88 <tr tal:define="name string:creation">
89 <th i18n:translate="">Creation Date:</th>
90 <td metal:use-macro="search_input"></td>
91 <td metal:use-macro="column_input"></td>
92 <td metal:use-macro="sort_input"></td>
93 <td metal:use-macro="group_input"></td>
94 </tr>
95
96 <tr tal:define="name string:creator;
97 db_klass string:user;
98 db_content string:username;"
99 tal:condition="db/user/is_view_ok">
100 <th i18n:translate="">Creator:</th>
101 <td metal:use-macro="search_select">
102 <option metal:fill-slot="extra_options" i18n:translate=""
103 tal:attributes="value request/user/id">created by me</option>
104 </td>
105 <td metal:use-macro="column_input"></td>
106 <td metal:use-macro="sort_input"></td>
107 <td metal:use-macro="group_input"></td>
108 </tr>
109
110 <tr tal:define="name string:activity">
111 <th i18n:translate="">Activity:</th>
112 <td metal:use-macro="search_input"></td>
113 <td metal:use-macro="column_input"></td>
114 <td metal:use-macro="sort_input"></td>
115 <td> </td>
116 </tr>
117
118 <tr tal:define="name string:actor;
119 db_klass string:user;
120 db_content string:username;"
121 tal:condition="db/user/is_view_ok">
122 <th i18n:translate="">Actor:</th>
123 <td metal:use-macro="search_select">
124 <option metal:fill-slot="extra_options" i18n:translate=""
125 tal:attributes="value request/user/id">done by me</option>
126 </td>
127 <td metal:use-macro="column_input"></td>
128 <td metal:use-macro="sort_input"></td>
129 <td> </td>
130 </tr>
131
132 <tr tal:define="name string:priority;
133 db_klass string:priority;
134 db_content string:name;">
135 <th i18n:translate="">Priority:</th>
136 <td metal:use-macro="search_select">
137 <option metal:fill-slot="extra_options" value="-1" i18n:translate=""
138 tal:attributes="selected python:value == '-1'">not selected</option>
139 </td>
140 <td metal:use-macro="column_input"></td>
141 <td metal:use-macro="sort_input"></td>
142 <td metal:use-macro="group_input"></td>
143 </tr>
144
145 <tr tal:define="name string:status;
146 db_klass string:status;
147 db_content string:name;">
148 <th i18n:translate="">Status:</th>
149 <td metal:use-macro="search_select">
150 <tal:block metal:fill-slot="extra_options">
151 <option value="-1,1,2,3,4,5,6,7" i18n:translate=""
152 tal:attributes="selected python:value == '-1,1,2,3,4,5,6,7'">not resolved</option>
153 <option value="-1" i18n:translate=""
154 tal:attributes="selected python:value == '-1'">not selected</option>
155 </tal:block>
156 </td>
157 <td metal:use-macro="column_input"></td>
158 <td metal:use-macro="sort_input"></td>
159 <td metal:use-macro="group_input"></td>
160 </tr>
161
162 <tr tal:define="name string:assignedto;
163 db_klass string:user;
164 db_content string:username;"
165 tal:condition="db/user/is_view_ok">
166 <th i18n:translate="">Assigned to:</th>
167 <td metal:use-macro="search_select">
168 <tal:block metal:fill-slot="extra_options">
169 <option tal:attributes="value request/user/id"
170 i18n:translate="">assigned to me</option>
171 <option value="-1" tal:attributes="selected python:value == '-1'"
172 i18n:translate="">unassigned</option>
173 </tal:block>
174 </td>
175 <td metal:use-macro="column_input"></td>
176 <td metal:use-macro="sort_input"></td>
177 <td metal:use-macro="group_input"></td>
178 </tr>
179
180 <tr>
181 <th i18n:translate="">No Sort or group:</th>
182 <td> </td>
183 <td> </td>
184 <td><input type="radio" name="@sort" value=""></td>
185 <td><input type="radio" name="@group" value=""></td>
186 </tr>
187
188 <tr>
189 <th i18n:translate="">Pagesize:</th>
190 <td><input name="@pagesize" size="3" value="50"
191 tal:attributes="value request/form/@pagesize/value | default"></td>
192 </tr>
193
194 <tr>
195 <th i18n:translate="">Start With:</th>
196 <td><input name="@startwith" size="3" value="0"
197 tal:attributes="value request/form/@startwith/value | default"></td>
198 </tr>
199
200 <tr>
201 <th i18n:translate="">Sort Descending:</th>
202 <td><input type="checkbox" name="@sortdir"
203 tal:attributes="checked python:request.sort[0] == '-' or request.sort[0] is None">
204 </td>
205 </tr>
206
207 <tr>
208 <th i18n:translate="">Group Descending:</th>
209 <td><input type="checkbox" name="@groupdir"
210 tal:attributes="checked python:request.group[0] == '-'">
211 </td>
212 </tr>
213
214 <tr tal:condition="python:request.user.hasPermission('Edit', 'query')">
215 <th i18n:translate="">Query name**:</th>
216 <td tal:define="value request/form/@queryname/value | nothing">
217 <input name="@queryname" tal:attributes="value value">
218 <input type="hidden" name="@old-queryname" tal:attributes="value value">
219 </td>
220 </tr>
221
222 <tr>
223 <td>
224
225 <input type="hidden" name="@action" value="regexp_search">
226 </td>
227 <td><input type="submit" value="Search" i18n:attributes="value"></td>
228 </tr>
229
230 <tr><td> </td>
231 <td colspan="4" class="help" i18n:translate="">
232 *: The "all text" field will look in message bodies and issue titles<br>
233 <span tal:condition="python:request.user.hasPermission('Edit', 'query')">
234 **: If you supply a name, the query will be saved off and available as a
235 link in the sidebar
236 </span>
237 </td>
238 </tr>
239 </table>
240
241 </form>
242
243 <disabled script language="javascript" type="text/javascript">
244 //<!--
245 onClick_RegExp();
246 //-->
247 <disabled /script>
248
249 </td>
250
251 </tal:block>
252 <!-- SHA: d7999f394badc861c290fe332cb72634191352fc -->
And here is the new action handler source code::
1 import cgi, types
2
3 # templating import required for roundup 1.3.2, at a minimum
4 # from roundup.cgi import templating
5 from roundup.cgi.actions import SearchAction
6 from roundup.cgi.exceptions import Redirect
7 from roundup.extensions import dbwrapper
8
9 class RegExpSearchAction(SearchAction):
10 def handle(self):
11 """\
12 Mangle some of the form variables.
13
14 Set the form ":filter" variable based on the values of the filter
15 variables - if they're set to anything other than "dontcare" then add
16 them to :filter.
17
18 Handle the ":queryname" variable and save off the query to the user's
19 query list.
20
21 Split any String query values on whitespace and comma.
22
23 """
24
25 self.fakeFilterVars()
26 queryname = self.getQueryName()
27
28 # Prepare for regular expression
29 if self.form.has_key('reg_exp'):
30 self.form.value.remove(self.form['reg_exp'])
31
32 if queryname:
33 error_message = '''Sorry, regular expression searches can't be saved yet'''
34 self.client.error_message.append(error_message)
35 self.client.template = 'search'
36 return
37
38 elif not self.form.has_key('@search_text'):
39 error_message = '''Missing regular expression in 'All text' field'''
40 self.client.error_message.append(error_message)
41 self.client.template = 'search'
42 return
43
44 else:
45 reg_exp = self.form['@search_text'].value
46
47 if self.form.has_key('re_ignorecase'):
48 re_ignorecase = int(self.form['re_ignorecase'].value)
49 else:
50 re_ignorecase = 0
51
52 else:
53 reg_exp = None
54
55 # editing existing query name?
56 old_queryname = ''
57 for key in ('@old-queryname', ':old-queryname'):
58 if self.form.has_key(key):
59 old_queryname = self.form[key].value.strip()
60
61 # handle saving the query params
62 if queryname:
63 # parse the environment and figure what the query _is_
64 req = templating.HTMLRequest(self.client)
65
66 # The [1:] strips off the '?' character, it isn't part of the
67 # query string.
68 url = req.indexargs_url('', {})[1:]
69
70 key = self.db.query.getkey()
71 if key:
72 # edit the old way, only one query per name
73 try:
74 qid = self.db.query.lookup(old_queryname)
75 if not self.hasPermission('Edit', 'query', itemid=qid):
76 raise exceptions.Unauthorised, self._(
77 "You do not have permission to edit queries")
78 self.db.query.set(qid, klass=self.classname, url=url)
79 except KeyError:
80 # create a query
81 if not self.hasPermission('Create', 'query'):
82 raise exceptions.Unauthorised, self._(
83 "You do not have permission to store queries")
84 qid = self.db.query.create(name=queryname,
85 klass=self.classname, url=url)
86 else:
87 # edit the new way, query name not a key any more
88 # see if we match an existing private query
89 uid = self.db.getuid()
90 qids = self.db.query.filter(None, {'name': old_queryname,
91 'private_for': uid})
92 if not qids:
93 # ok, so there's not a private query for the current user
94 # - see if there's one created by them
95 qids = self.db.query.filter(None, {'name': old_queryname,
96 'creator': uid})
97
98 if qids:
99 # edit query - make sure we get an exact match on the name
100 for qid in qids:
101 if old_queryname != self.db.query.get(qid, 'name'):
102 continue
103 if not self.hasPermission('Edit', 'query', itemid=qid):
104 raise exceptions.Unauthorised, self._(
105 "You do not have permission to edit queries")
106 self.db.query.set(qid, klass=self.classname,
107 url=url, name=queryname)
108 else:
109 # create a query
110 if not self.hasPermission('Create', 'query'):
111 raise exceptions.Unauthorised, self._(
112 "You do not have permission to store queries")
113 qid = self.db.query.create(name=queryname,
114 klass=self.classname, url=url, private_for=uid)
115
116 # and add it to the user's query multilink
117 queries = self.db.user.get(self.userid, 'queries')
118 if qid not in queries:
119 queries.append(qid)
120 self.db.user.set(self.userid, queries=queries)
121
122 # commit the query change to the database
123 self.db.commit()
124
125 # check for regular expression search
126 if reg_exp and self.classname == 'issue':
127 # not needed anymore
128 if self.form.has_key('@search_text'):
129 self.form.value.remove(self.form['@search_text'])
130
131 # open dbwrapper
132 dbw = dbwrapper.DBWrapper(self.db)
133
134 if re_ignorecase:
135 flags = dbwrapper.IGNORECASE
136 else:
137 flags = 0
138
139 flags += dbwrapper.LOCALE
140
141 issues = [ issue.id for issue, res in dbw.issue.re.search(reg_exp, flags, 'title') ]
142
143 flags += (dbwrapper.MULTILINE + dbwrapper.DOTALL)
144
145 messages = dbw.msg.re.search(reg_exp, flags, 'content')
146
147 results = dbw.issue.find(messages=messages)
148
149 for issue in results:
150 if not issue.id in issues:
151 issues.append(issue.id)
152
153 if self.form.has_key('id'):
154 if isinstance(self.form['id'], types.ListType):
155 form_issues = []
156
157 for minifield in self.form['id']:
158 if minifield.value:
159 form_issues += re.split('[+, ]', minifield.value)
160 else:
161 form_issues = re.split('[+, ]', self.form['id'].value)
162
163 self.form.value.remove(self.form['id'])
164
165 found_issues = []
166 for issue_id in form_issues:
167 if int(issue_id) in issues:
168 found_issues.append(int(issue_id))
169
170 issues = found_issues
171
172 if issues:
173 issues.sort()
174 else:
175 issues.append(-1)
176
177 self.form.value.append(cgi.MiniFieldStorage('id', '+'.join( [str(nodeid) for nodeid in issues] )))
178
179 args = []
180 for column in self.form.keys():
181 if isinstance(self.form[column], types.ListType):
182 values = []
183
184 for minifield in self.form[column]:
185 if minifield.value:
186 values.append(minifield.value)
187 else:
188 values = [self.form[column].value]
189
190 args.append('%s=%s'%(column, ','.join(values)))
191
192 raise Redirect, '%s%s?&%s'%(self.base, self.classname, '&'.join(args))
193
194
195 def init(instance):
196 instance.registerAction('regexp_search', RegExpSearchAction)