Is there a way to batch recalculate all nodes?
gemini - November 24, 2007 - 19:02
| Project: | Computed Field |
| Version: | 6.x-1.0-beta2 |
| Component: | Documentation |
| Category: | support request |
| Priority: | normal |
| Assigned: | Unassigned |
| Status: | active |
Description
I have a couple of thousands records and just added a computed field to their content type. How do I batch the calculation process for all the nodes? It doesn't seam to work on cron run, but it calculates the data on saving node. I would like to use the calculated feed as sorting criteria in the views, and in order to that I need somehow to recalculate values for all my nodes.
..any ideas?
#1
I figured it out... it is actually very simple - just pull the necessary records from the DB with SQL query and use node_save() in the loop.
Here are the 4 main lines of my code:
<?php$res = db_query("SELECT n.nid FROM {node} n WHERE n.type = 'your_content_type'");
print '<ol>';
while ($n = db_fetch_object($res)) {
node_save(&$n);
print '<li>'.l($n->title, 'node/'.$n->nid).'</li>';
}
print '</ol>';
?>
If anyone has large dataset - you might want to use LIMIT and process 100 nodes at the time or something along those lines. It think it would be good to add such function to the module as a manual recalculation.
#2
I agree. Seems like a fairly simple feature addition that could be useful to many people.
#3
I pulled your php into a module for my own use, here it is if anybody likes it.
#4
Hi Ted,
Just a short thanks & thumbs up for the helpful module. Worked like a charm.
Regards,
Wouter
#5
I tried this as part of the header of a view.
Two problems:
1) every node recomputed is then marked as "updated". Is there a way of forcing a recomputation but not making the node as updated?
2) the view itself doesn't update to reflect the changes till the next time. Is there a way of updating the view query in the header?
Many thanks in advance! This is a great CCK field.
#6
Following the advice in http://muzso.hu/node/4332 I think I've solved the updating bit ... now need to fix the reloading of the view query.
The Following code is in the header of my view (set to PHP input filter):
<?php// The calculated values are only recalculated when the node is
// loaded so force a reload now
$res = db_query("SELECT n.nid FROM {node} n WHERE n.type = 'session'");
print '<p>DEBUG: updated the following</p>';
print '<ol>';
while ($n = db_fetch_object($res)) {
// get the old timestamps before we save
$node_changed = $n->changed;
$rev_timestamp = $n->revision_timestamp;
node_save(&$n);
// undo changes in timestamp and changed flag
db_query("UPDATE {node} SET `changed` = '%d' WHERE `nid` = '%d'", $node_changed, $n->nid);
db_query("UPDATE {node_revisions} SET `timestamp` = '%d' WHERE `nid` = '%d' AND `vid` = '%d'", $rev_timestamp, $n->nid, $n->vid);
print '<li>'.l($n->title, 'node/'.$n->nid).'</li>';
}
print '</ol>';
?>
#7
subscribing.
#8
Is there a possibility of setting it as a action & trigger such that when a specific node gets updated/ created, this functionality of the computed fields being recomputed happens?
http://drupal.org/node/324704
#9
I've changed a little code as in #1 as follow:
<?php$res = db_query("SELECT n.nid FROM {node} n WHERE n.type = 'some_content'");
while ($n = db_fetch_object($res)){
$node = node_load($n->nid);
$n = node_save($node);
}
?>
So now it's working for me fine in Drupal 6.x (before not).
#10
IMHO updating alll nodes is like killing a fly with a tank... It's way too heavy :D
I think we should have a mechanism to avoid some computed field being cached by Drupal cache mechanism, but I'm not really sure if it's currently possible...
See http://drupal.org/node/332200
#11
Thanks for the update. I installed the module from .zip file in #3 and I was also getting
I changed the module posted by blookslic in #3 above based on code in #9 above and the messages are gone.
See attached revised .zip (use at your own risk)
#12
the regenerate module from post 11 says
This version is incompatible with the 6.7 version of Drupal core.
This would help me with this issue correct?
http://drupal.org/node/343786#comment-1175338
#13
Views Bulk Operations can do this correct?
#14
I'm running Drupal 6.8 and it worked fine for me. I wonder if it could be a PHP version issue for you. You could try upgrading to 6.8 and see what happens.
#15
Views Bulk Operations did exactly what I needed.
#16
How about time out issues? Is there a way to have this run with cron, like the search indexing? I can imagine a lot of time needed on large sites.
@#10:
I have computed field which can take advantage of other modules. If they are enabled after some content has been created, then that content will not have the correct links and features, and since I update it infrequently, some sites might have an old version, which will be updated in the future, when the development is in a stable state.
Tnx for the module, I will try it out.
#17
@#12 I added
core = "6.x"into regenerate.info, but still it doesn't work, problem with an extra argument in regenerate_menu(), but after fixing that, still no joy.
There is a define for REGENERATE_UPDATE_NODE_TYPES, which is never used in the file, I tried changing the places where UPDATE_NODE_TYPES was used to see if that was a problem, but no.
I am stumped...
#18
I spent far too much time this night trying to get this to work. I got it to finally show the form, but it seems that on form submit it doesn't call the right handler.
If anyone wants to try fixing the remaining problem with the form submit, this is were you should start on 6.8
I should have just gone with views and custom code.
#19
kenorb (comment #9),
I need help with your code. When I enter the code in the header of my view it does not work, In the header I can see a piece of the code
nid); $n = node_save($node); } ?>
and nothing happens (meaning it does not recalculate the computed field,
can you help me figure out what I am doing wrong? . The only thing that I changed from your code was the content type.
Thanks,
#20
@finlaycm: Make sure that you have enabled PHP Filter and the input format is in PHP, then it should work.
#21
Subscribing
#22
I think a nice way for keeping the computed column data in synch with the computed value would be:
1) On Node view, the module should retrieve the stored value and the computed value and always display the computed value.
2) If they are different, then update the stored value.
3) Never cache the computed columns with the page view if that is possible.
This to me seems like it would be best for most use cases. If a user goes from a node listing page that has stale stored computed data to the Node Details page, they would always get the up to date info when it really mattered and the corresponding row in the listing view would then get updated. If a computation is to expensive for a node page you could opt out of the showing the computed value on the node page.
What do you think? If people are interested I could put together a patch maybe.
(Also, note in the Display Fields setting if you set it to "Computed Value" for the node then it still shows the database value. At least for me. I thought that would take care of #1 above but it did not.)
#23
Great Idea.
@subscribing
#24
subscribing
#25
http://drupal.org/node/332200
#26
Fixed spelling, so that if someone searches for "recalculate" they'll find this issue.
#27
Hi to all,
I'm working in D6.10 and computed field 6.x-1.0-beta2 and it seem that I found a solution thats it's working for me.
I've
- a date field named "field_due_date"
- computed field named "field_check_date" that check if date < today
and I created the following trigger and actions:
-Rule setting: event = "Content is going to be viewed"
-Condition: Viewed content is *the name of my content type*
-Actions: Populate viewed content's field "field_check_date"
Show a configurable message on the site (only to check that the rule is working)
And thus, any time my content type is going to be viewed, the rule populate the value of the field "field_check_date" with the Computed Code and the Display format of the computed field.
Only, i need to check tomorrow (when the date change) if this is still working.
Regards,
Jorge
#28
You can easily do this with Views Bulk Operations. Install the module and go to the views page, where VBO comes with a default view that mimics the content listing page of Drupal. Instead, you can select all nodes at once (eg of a certain type), and you should choose "Execute arbitrary PHP script". In the body, write
node_save($object);. This will resave all the nodes on your website and thus trigger the calculation of the computed fields!#29
subscribed
Hey why not make the 'regenerate' script into a full blown module?
#30
I think this would make more sense as a part of the Computed Field module, which would expose a filter to recalculate computed field nodes (by re-saving the node). VBO is just a way to select more than 25 nodes...
#31
The real solution would have to look something like this:
- Calculate scores on node creation
- Cache scores in a cache table
- Set cache duration, e.g. 1 hour - 1 day - 1 week
- Check cache period on node-view
- If the cache period expired, re-calculate scored for that node
This 1. automates the entire process, 2. takes care of performance issues - we only recalculate on node-view once per cache lifetime. Which is fine. (Search engines visiting our sites will update all nodes ;=) )
I'll probably have to invent this myself, because I heavily rely on computed fields, which users can change with editablefields.
#32
The idea in #28 has a gotcha--VBO doesn't clear the data residue from a prior node before processing the next node. I have filed #564442: Data residue/namespace not cleared between bulk node operations in the Views Bulk Operations issue queue.
#33
Subscribe
#34
subscribing, thx
#35
Well Jorge,
Did it work? I am now trying to use your trick. And it seems I only get the confurable message when I go in and view a node (not view a view) as I was hoping.
So I have to go to a node of the particular type. Then Drupal displays my message. But the computed field does not get updated...
Any ideas?
Thanks
Morten
#36
subscribing!
#37
#28 Solved it for me. Thanks.
#38
I'd still like to see a module-integrated solution to this. I have computed field values that are based off of other computed field values. With solution #9 and #1 it takes two reloads to get the numbers right. That's very db-intensive.
The same applies to solution #28, which is even worse because it reloads all nodes.
I have 30 content types. If a user changes one field, a calculated value may change. Only calculated fields that are accessing this particular piece of data should be updated. Again, keep in mind that there are calculated fields that access data off of other calculated fields.
Ideas anyone?
#39
I found a solution to this (#38) particular problem. It's not very elegant, but it works.
I use Calculated Fields to calculate the first tier value, i.e. x = 2 + 4, y = 1 + 2
I use Views Custom Field to calculate the second tier value that is based of off two "first tier calculations", i. e. z = x + y, z = 2 + 4 + 1 + 2 = 9.
Again, I would love to use only one of those two solutions, but I don't have a choice.
The clear disadvantage of Views Custom Field is that it doesn't store its values in the db.
#40
I have a problem running any of the given code snippets above because I have a class declared inside the textarea of the computed field PHP code. So I get a fatal error: Cannot redeclare class [MYCLASS] in [...] computed_field.module(161) : eval()'d code on line 921
I tried to take the code outside and save it in a file but then add a problem of re-declaring a function (http://drupal.org/node/623144).
Is there any way I can do it nevertheless?
Thanks!