bookmark.db

George2005 escribió:
Well, both suggestions seem OK, there is a one problem in general (as you wrote before),
ie if some recordings were stored in original bookmark.db then they were lost by replacing the
modified bookmark.db file - sugested solution follows:
I noticed that if bookmark.db is missing in /usr/local/etc/dvdplayer/ at the time of starting then
DvdPlayer creates "fresh" bookmark.db, which is completely empty (without any tables except of
master one) and it is exactly 1024 bytes long. I tried to replaced it with modified bookmark.db
and it seemed to be OK. So it could be possible to detect if such file exists with filesize
of 1024 B and if yes, then replace it by modified bookmark.db. If bookmark.db wouldn't exist at all
then PM could copy modified.db without any problems. And if bookmark.db would exist with filesize
more than 1 KB and wouldn't contain GUID then it probably stores some recordings so they could be
lost by replacing bookmark.db. So there would be a special button in LGMenu which could "upgrade"
bookmark.db to modified version. In fact we could use php sqlite api to create tHarmony
table & triggers in original bookmark.db. Of course there could be an option in PM.ini to
allow "force" the replacing original bookmark.db by modified version automatically even if it
isn't empty.

Hello, thanks for the new ideas.

*1* Initial bookmark.db

I have deleted actual db in lg, and then restarted, but no initial .db is created,
i have looked in both places.

I have only these files:
root@Venus:/usr/local/etc/dvdplayer# ls
disclosureAgree.dat schedule_history.db setup.real
dtv_channel.txt schedule_record.db youtubeKey
isRTC script

It seems for ,any reasons, it is creating it for some equipments, and not for others
In my cases, it is created when I stop playing a file.
Then I have checked this file has 17kbytes.

Could you please send me this initial bookmark.db of 1024bytes ?


*2* Replacement.
Yes, good idea.
I think that when PM starts, will check, and if file does not exists or size is 1024B,
will be replaced by our bookmark.db , which we could have a safe copy in the firmware,
eg in: /usr/local/firmware/bookmark.db (Is that good place Keltek

Do you think it is good to copy in both places: /usr/local/etc/dvdplayer and /tmp/ramfs ?


And that's all folks; I will not worry for the other cases,
because if a person is installing a fresh firmware in lg, then all original db of users
will be lost anyway.

And if not installed by firmware but PM upgrade (num.code 987), there is an install script,
that it is first downloaded and then executed, and we can do something like
sqlite3 /usr/local/etc/dvdplayer/bookmark.db < install_triggers.sql

Or let the user losts all the restart point, I mean, this is not crucial information,
and is going to pass only once; we can info the user and if he agree then install the new
features

evr escribió:

Keltek escribió:[[...]]I hesitate for a long time to react or not... so here is my opinion. [...]

Keltek,
Thanks for finally deciding to react! [...] I will reply as soon as I find some time to think again about this hobby (I'v just entered in a too-busy period at work and this will last until the end of January)[...]

Well, not end of January yet, but I found some time today.

For me it is completely clear that the "Hello-Wait-reboot" loop and other hangups are produced because of some of the modifications to the original LG firmwares. I used the official firmware MS400_091209_0118.img and MS400_101027_0128.img for more than one month without any problem; then I decided to test Harmony and in less then 24 hours I got two "Hello-wait-reboot" oops. After the second reisntall, I completely stopped all the extra services of Custom-KV and this lasted for 3 days before the next "Hello-Wait-reboot" loop. Then I installed this special firmware (official firmware + PM3.5beta -- no Custom-KV) and, after 10 days, yesterday night, the "Hello-Wait-reboot" loop happened again (this morning I have just gone back to strictly official LG firmware MS400_101027_0128.img to see how long it works O.K.).

Of course I understand and appreciate your (and Victor's) efforts and, as I've said several times, I am wishing to help to the extent my scarce free time permits. I have been working before in several open-source projects (for embedded hardware) and I do have experience with the kind of difficulties entailed by machines such as the MS400/450. I know that 99% of "strange" troubles finally boil down to exhausting the scarce resources (power, memory and flash, typically -- of course, power is not that much an issue with the MS400/450). So I am this 99% convinced that this is also the case with our hangups.

256Mbyts is very little memory for a program like DvdPlayer, which, in theory could reach more than 800Mbyts of VM demand, according to what you can reed in /proc/<pid_of_DvdPlayer>/status (fortunately this seems to never happen, but without seing the source code, who knows...). Then comes the swap, which is strangely divided into a ridiculous part in flash and another also too-small amount in the part2 of the hard disk. I must say that from my three "Hello-Wait-reboot" oops with Harmony, at least the second one happened after I powered off the machine with about 12kBytes of swap used according to /proc/swaps. So swap usage might be somehow in the root of the problem. In my experience, embedded hardware and swap often are not good friends; so I see `swap usage > 0' as a suspicious symptom (and more so as I do not understand at all what is the purpose of the 32Mbyts flash swap!).

Regarding the SF.net tracking system, I have seen it but it does not look to be really commonly used in practice. I do not know how many people does some coding in this project -- may be three or four (or five if you count me ;), but it does not seem that the info flows through the tracking system but through this forum (and probably PMs too) instead. Personally I do prefer SF communication tools, ideally including a developers SF mailing list, which I find much more comfortable than the web-based way of communication of this forum. But we should agree in this so we do not have to repeat messages and we do not miss what is important.

Finally, thanks for the links to your toolchain. I wil download the things I still miss and start trying in February.

Cheers,

vic1972 escribió:
Hello, thanks for the new ideas.

*1* Initial bookmark.db

I have deleted actual db in lg, and then restarted, but no initial .db is created,
i have looked in both places.

I have only these files:
root@Venus:/usr/local/etc/dvdplayer# ls
disclosureAgree.dat schedule_history.db setup.real
dtv_channel.txt schedule_record.db youtubeKey
isRTC script

It seems for ,any reasons, it is creating it for some equipments, and not for others
In my cases, it is created when I stop playing a file.
Then I have checked this file has 17kbytes.

Could you please send me this initial bookmark.db of 1024bytes ?

It's strange that you couldn't get the empty 1KB bookmark.db, because it creates me regularly after deleting previous bookmark.db and restarting ... but as you know the MS4x0H is a quite mysterious machine
Empty 1KB bookmark.db from my machine is here: http://www.rysi.duha.cz/lg/empty_bookmark/bookmark.db

vic1972 escribió:
*2* Replacement.
Yes, good idea.
I think that when PM starts, will check, and if file does not exists or size is 1024B,
will be replaced by our bookmark.db , which we could have a safe copy in the firmware,
eg in: /usr/local/firmware/bookmark.db (Is that good place Keltek

Do you think it is good to copy in both places: /usr/local/etc/dvdplayer and /tmp/ramfs ?

OK, as far as I know, /tmp/ramfs/bookmark.db doesn't exist just after machine starting, it is copied/modified just at the moment of the first playing interruption. So at the moment of starting it should be enough to copy it into /usr/local/etc/dvdplayer.

vic1972 escribió:
And that's all folks; I will not worry for the other cases,
because if a person is installing a fresh firmware in lg, then all original db of users
will be lost anyway.

And if not installed by firmware but PM upgrade (num.code 987), there is an install script,
that it is first downloaded and then executed, and we can do something like
sqlite3 /usr/local/etc/dvdplayer/bookmark.db < install_triggers.sql

Or let the user losts all the restart point, I mean, this is not crucial information,
and is going to pass only once; we can info the user and if he agree then install the new
features

OK, I agree, we can import triggers into existing bookmark.db using sqlite3 automatically without confirmation, it should preserve stored recordings anyway ...

Very good,
so when needed I will execute the following:
/tmp/hdd/root/custom/bin/sqlite3 /usr/local/etc/dvdplayer/bookmark.db < /usr/local/etc/dvdplayer/triggers.sql

I have just tested that above line, and is fine!!!

triggers.sql is your work
http://dl.dropbox.com/u/684543/varios/ms450h/RePlay/triggers.sql

I have done a basic testing of your triggers and seems to be working fine, more testing is needed.
I will modify PM for the necessary checkings.
So, I think we found the best solution, instead of removing actual bookmark.db,
we are loading dinamically the triggers and temp, table. ok in the actual bookmark.db


Some bad news,
I have just realized that when a recording is active, the actual file been saving is actually the first in the REC list;
So, probably we will have to recommend the alphabetical order, in order that 0_RecentFile be always the first.

vic1972 escribió:Very good,
so when needed I will execute the following:
/tmp/hdd/root/custom/bin/sqlite3 /usr/local/etc/dvdplayer/bookmark.db < /usr/local/etc/dvdplayer/triggers.sql

I have just tested that above line, and is fine!!!

triggers.sql is your work
http://dl.dropbox.com/u/684543/varios/ms450h/RePlay/triggers.sql

I have done a basic testing of your triggers and seems to be working fine, more testing is needed.
I will modify PM for the necessary checkings.
So, I think we found the best solution, instead of removing actual bookmark.db,
we are loading dinamically the triggers and temp, table. ok in the actual bookmark.db

Great!!! I also think that it could be the best thing to import our sql statements into original bookmark.db instead of overwriting it.

vic1972 escribió:
Some bad news,
I have just realized that when a recording is active, the actual file been saving is actually the first in the REC list;
So, probably we will have to recommend the alphabetical order, in order that 0_RecentFile be always the first.

Well, it's the same situation as I noticed too, but as I wrote before the sorting by LATEST also could be suitable in case that the first item is the actual last played file.


Btw: I am thinking about extending the triggers' functionality for all file types and directories - for the path parsing it could be great to use regular expressions but the regexp function is not implemented in sqlite3 by default so it is needed to load it as an extension. I found only this: https://github.com/eatnumber1/sqlite-regexp, which uses pcre to handle regexp. Do you think that it is possible to compile it?

Hello,
regarding sqlite-regexp, lets ask Keltek
it seem a single .c file of 82 lines, very simple,
what i am not sure, what to do next, include as a module ?....

vic1972 escribió:Hello,
regarding sqlite-regexp, lets ask Keltek
it seem a single .c file of 82 lines, very simple,
what i am not sure, what to do next, include as a module ?....

OK, thanks.
Btw: It is a sqlite extension so is it is loaded by load_extension() function directly from SQL code, but as there is a discussion between Keltek and evr about resources exhausting etc, I am not sure what it will do it exactly if I want to load it inside the bookmark.db - I don't want to cause to DvdPlayer to load it repeteadly during the bookmark.db accessing to finally exhaust resources, memory etc.

So, Keltek, can you help please ?

This is hard to answer
I don't know to much about sqlite. There can be a limitation of sqlite used in DvdPlayer, which means all of our binaries (php and samba for now) use the shared library libsqlite3.so (which is almost 1MB in size) but DvdPlayer use the statically compiled-in unknown version which can or can't support the extended modules. This must be tested.
Now we are unable to use shared library in PipeManagement, because this library sitting on /tmp/hdd/root partition which is mounted by DvdPlayer and we are unable to copy this library to place we can use (like to /lib or /usr/lib) because it is very very big.
So you should compile the extension module and put it to /usr/local/etc and in first step, try simple usage of this extension (using the sqlite3 binary) and then use it in trigger. Try to load the extension twice - what happen?
I look at the source (I hope I have a right url https://github.com/eatnumber1/sqlite-regexp ) and see, there is dependency on another library - libpcre - which will do all the stuff around the regexp so be prepared to make this library in public place for DvdPlayer.

ok,
I have committed all the necessary changes.
Now there is detection of the GUID, if not, then triggers are installed.

It is available for testing.
It is courious, but the first time is not working properly, I will have to check the reason.
If you could have a test...

vic1972 escribió:ok,
I have committed all the necessary changes.
Now there is detection of the GUID, if not, then triggers are installed.

It is available for testing.
It is courious, but the first time is not working properly, I will have to check the reason.
If you could have a test...

Thanks a lot. I tested it and I get the same results - there was not OK for the first time.

During testing I deleted bookmark.db from both places and reboot. After starting up the new bookmark.db created but with 7KB filesize which contains only master_table, tHarmony and one trigger only, so it could be the "initial" 1KB bookmark.db with partial modification - other triggers couldn't be created because tParameter table has not existed yet!

So I think that the problem is that the importing the SQL statements occurs too early so it partially (or completely) fails. I will thinking about it more but the one solution could be to test if existing bookmark.db already contains tParameter table (either by binary searching or using sqlite3) and if yes then import SQL statements.

Keltek escribió:This is hard to answer
I don't know to much about sqlite. There can be a limitation of sqlite used in DvdPlayer, which means all of our binaries (php and samba for now) use the shared library libsqlite3.so (which is almost 1MB in size) but DvdPlayer use the statically compiled-in unknown version which can or can't support the extended modules. This must be tested.
Now we are unable to use shared library in PipeManagement, because this library sitting on /tmp/hdd/root partition which is mounted by DvdPlayer and we are unable to copy this library to place we can use (like to /lib or /usr/lib) because it is very very big.
So you should compile the extension module and put it to /usr/local/etc and in first step, try simple usage of this extension (using the sqlite3 binary) and then use it in trigger. Try to load the extension twice - what happen?
I look at the source (I hope I have a right url https://github.com/eatnumber1/sqlite-regexp ) and see, there is dependency on another library - libpcre - which will do all the stuff around the regexp so be prepared to make this library in public place for DvdPlayer.

Thanks a lot for your information. It's clear that DvdPlayer uses the statically compiled-in sqlite3 to work with db files. But it seems that it doesn't support extension loading actually. As the names of functions etc. are shown in plain text I searching DvdPlayer in binary for "load_extension" and I found nothing. I did the same with sqlite3 and I found it. So I have to avoid using regexp and use built-in string functions which are not quite good but I hope that result will work, however by using not so smart code as the regexp could be ...

YESSS
I see.
So, besides of checking of the GUID, I will also check that tParameter exists;
then I will apply the patch with the triggers.
But anyway, I guess we will still lost the first play (but anyway, I think this is fine will test .
Probably it will be changed second time LG boots...

AHH one more thing, I lightly changed your triggers, adding id when inserting in the tHarmony table.
So, if the patch for installating triigers is repeated (for any reason), the table will not grow.

Código:


INSERT INTO tHarmony (id, fKey, fValue) VALUES (1,'GUID','DC1E300C-E7F1-4EF9-B5BF-40EDB6DC56C6'); /* unique Id in TEXT (STRING) form for identification the modified db file */
INSERT INTO tHarmony (id, fKey, fValue) VALUES (2,'RecentFilenameText','fileset:///tmp/ramfs/volumes/HDD ( Internal )/REC/0_RecentFile.ts'); /* full name od Recent File in TEXT form */
INSERT INTO tHarmony (id, fKey, fValue) VALUES (3,'RecentFilenameBLOB',X'66696C657365743A2F2F2F746D702F72616D66732F766F6C756D65732F484444202820496E7465726E616C20292F5245432F305F526563656E7446696C652E7473'); /* full name od Recent File in BLOB form */
INSERT INTO tHarmony (id, fKey, fValue) VALUES (4,'RecentFilenameNumber','9999'); /* the number of Recent File in db records */
INSERT INTO tHarmony (id, fKey, fValue) VALUES (5,'AutoRecoveryEnable',1); /* 1 - enable Auto Recovery of Recent File db records */
INSERT INTO tHarmony (id, fKey, fValue) VALUES (6,'AutoRecoveryRaiseTrigger',zeroblob(16)); /* used for raise the recovery trigger by changing a value to a new value */


Looking at how well you did with the triggers, I guess you are more than able to do anything without extensions ...

vic1972 escribió:YESSS
I see.
So, besides of checking of the GUID, I will also check that tParameter exists;
then I will apply the patch with the triggers.
But anyway, I guess we will still lost the first play (but anyway, I think this is fine will test .
Probably it will be changed second time LG boots...

AHH one more thing, I lightly changed your triggers, adding id when inserting in the tHarmony table.
So, if the patch for installating triigers is repeated (for any reason), the table will not grow.

Código:


INSERT INTO tHarmony (id, fKey, fValue) VALUES (1,'GUID','DC1E300C-E7F1-4EF9-B5BF-40EDB6DC56C6'); /* unique Id in TEXT (STRING) form for identification the modified db file */
INSERT INTO tHarmony (id, fKey, fValue) VALUES (2,'RecentFilenameText','fileset:///tmp/ramfs/volumes/HDD ( Internal )/REC/0_RecentFile.ts'); /* full name od Recent File in TEXT form */
INSERT INTO tHarmony (id, fKey, fValue) VALUES (3,'RecentFilenameBLOB',X'66696C657365743A2F2F2F746D702F72616D66732F766F6C756D65732F484444202820496E7465726E616C20292F5245432F305F526563656E7446696C652E7473'); /* full name od Recent File in BLOB form */
INSERT INTO tHarmony (id, fKey, fValue) VALUES (4,'RecentFilenameNumber','9999'); /* the number of Recent File in db records */
INSERT INTO tHarmony (id, fKey, fValue) VALUES (5,'AutoRecoveryEnable',1); /* 1 - enable Auto Recovery of Recent File db records */
INSERT INTO tHarmony (id, fKey, fValue) VALUES (6,'AutoRecoveryRaiseTrigger',zeroblob(16)); /* used for raise the recovery trigger by changing a value to a new value */


Looking at how well you did with the triggers, I guess you are more than able to do anything without extensions ...

OK, adding "id" values is very good! In fact I wasn't thinking about the repeated importing at the moment of creating the original version of triggers ... So we can add "IF NOT EXISTS" clauses to all CREATE commands to have a full corect code ...

Hello,
new version is available for testing.

Now, strange chars are scaped; and triggers inserted when needed.

I need some help, because i do not know why, bookmark.db is not been updated by the triggers.
Do not know what i have done wrong; if you could have a look. thanks

I have even detected some strange behaviour,
selecting a film.ts (not the 0_RecentFile.ts), stoping it, and playing again film.ts, it is not resuming well, it is resuming in a different
time ... very weird Like getting the resuming info of a different file...

vic1972 escribió:I have even detected some strange behaviour,
selecting a film.ts (not the 0_RecentFile.ts), stoping it, and playing again film.ts, it is not resuming well, it is resuming in a different
time ... very weird Like getting the resuming info of a different file...

Well, you are right, the problem was that bookmark.db was not updated at all even in case of "normal" ts file. I notice that no db records were created for Recent file because autorecovery trigger was not raised yet. Maybe it is caused by SQL errors occuring due the repeated importing of triggers.sql. So I fixed it by adding "IF NOT EXISTS" clauses to CREATE statements and IGNORE clauses (which are ON CONFLICT IGNORE clauses actually) to INSERT statements for tHarmony records. I also added the statement to force raising the autorecovery trigger at the end of triggers.sql. I tested it briefly and it seemed that it could work - bookmark.db is updated both for "normal" ts file as well as for Recent file. A new version of triggers is here: http://www.rysi.duha.cz/lg/triggers/triggers.sql

OK thanks,
I will test this new triggers tonight.
Btw, I have just uploaded the new stuff in case you want to test also.

Feeling like this is the final attempt , and will work fine


OK, I have noted in the new triggers, that the id are inserted for tHarmony are inserted, fine.
and one question, i have noted at the end, there is a sentence after the end command, is that fine?

Código:


...
END;

/* force to raise AutoRecoveryHandle trigger to create initial db records for Recent File */
UPDATE tHarmony SET fValue = randomblob(16) WHERE fKey = 'AutoRecoveryRaiseTrigger';


vic1972 escribió:OK thanks,
I will test this new triggers tonight.
Btw, I have just uploaded the new stuff in case you want to test also.

Feeling like this is the final attempt , and will work fine


OK, I have noted in the new triggers, that the id are inserted for tHarmony are inserted, fine.
and one question, i have noted at the end, there is a sentence after the end command, is that fine?

Código:


...
END;

/* force to raise AutoRecoveryHandle trigger to create initial db records for Recent File */
UPDATE tHarmony SET fValue = randomblob(16) WHERE fKey = 'AutoRecoveryRaiseTrigger';


Well, it is correct, actually, because the END command which you mentioned belongs to AutoRecoveryHandle trigger code. The last UPDATE command just forces raising this AutoRecoveryHandle trigger so it must be outside the trigger code.

yes, you right, of create triggers: AutoRecoveryHandle. fine.

More things,
in Spanish we have the ñ and the accents áéíóú and then chars like ()[] _ and space.
Hope this will work also for your language, I have escaped all chars that are not numbers , a-z and A-Z
so: file like Hola camión.ts will be treated as:
ln -f Hola\ cami\ón.ts 0_RecentFile.ts
Those are woking fine here.
Is this working fine for you with your language? Can you paste a sample of filename in Czech

vic1972 escribió:yes, you right, of create triggers: AutoRecoveryHandle. fine.
More things,
in Spanish we have the ñ and the accents áéíóú and then chars like ()[] _ and space.
Hope this will work also for your language, I have escaped all chars that are not numbers , a-z and A-Z
so: file like Hola camión.ts will be treated as:
ln -f Hola\ cami\ón.ts 0_RecentFile.ts
Those are woking fine here.
Is this working fine for you with your language? Can you paste a sample of filename in Czech

Well, in the Czech language there are a lot of characters with accents: ěščřžýáíéóúůňďť & their uppercase variants ...
For filename samples in Czech please see this post with bookmark.db dump - there are a lot of them there.

Congratulations!!!!
Preliminary testing are fine. Ready to download with secret numeric code.
This could be the end of weeks of work and investigations

vic1972 escribió:Congratulations!!!!
Preliminary testing are fine. Ready to download with secret numeric code.
This could be the end of weeks of work and investigations

It sounds great!

But maybe we have to solve another problem. I downloaded the last PM and tested it just in the "real" conditions, ie I programmed a timer and then played some ts file. When the programmed time occured, warning window was shown and after 30 seconds machine returned to TDT mode. So I tried to push the Red button but it didn't worked (as we discussed about it before) so I went to REC menu manually and I saw that the current active recording is at the first place of list instead of 0_RecentFile.ts! Later I tried to start the recording manually and the active recording was also placed as the first item in the list of recordings. I also noticed very interesting thing - active recording was ALWAYS at the first place in the list regardless the type of sorting! So the changing the sorting method didn't matter and active recording with name of "Kriminálka Paříž ...." was in the list before 0_RecentFile.ts even using the sorting by TITLE NAME.

Please could you test it, too? If you will get the same results then we should modify the Red Button command sequence slighty - in case of active recording we must add the "X" command ("arrow down") just after the "$" command to select the second item in the list (ie 0_RecentFile.ts) ...

This seems like endless history.
AH yes, I see what you mean, I cannot test right now, as i am away.
But yes, when a recording is active, it is created like a part_R000 like file, and it is always "touched" by the system,
so, you are right, it is always the first.
I will think of a solution...
think i have already.
When we are in the list of recording, you know there is a preview in little window, and some log is generated with the filename;
so I can sniffer a bit (yes i know;) , and detect if it is 0_RecentFile.ts and if not move one hit down as you suggested. good idea.

AHH One more thing, yesterday I detected that the red button in the remote controller is labelled
curiouly as Repeat. funny right

This color-button was selected because of the next available color button

Do you think it is a good idea, to change the name of the file from 0_RecentFile.ts to 0_ReplayFile.ts
or 0_Replay.ts .For me it is fine like it is now, but the idea is there ...

vic1972 escribió:AHH One more thing, yesterday I detected that the red button in the remote controller is labelled
curiouly as Repeat. funny right

This color-button was selected because of the next available color button

Do you think it is a good idea, to change the name of the file from 0_RecentFile.ts to 0_ReplayFile.ts
or 0_Replay.ts .For me it is fine like it is now, but the idea is there ...

That's great with the Red Button label And it is very good idea to rename RecentFile to Replay! I was also thinking about remove "0" from the start of name because the "future compatibility", ie replaying all movie files (not only ts), as I have some files with name which start with non-alphabetical character (eg. "!movie.avi"). But maybe it is my problem only so I don't insist of it