Page 1 of 3 123 LastLast
Results 1 to 15 of 31
  1. #1
    Member

    Join Date
    Nov 2010
    Posts
    31
    Thanks
    10
    Thanked 24 Times in 10 Posts

    New Python script to scrape KingOfSat for populating CCcam.channelinfo & oscam.srv

    Just thought I'd announce the development of a small tool I've been working on before I release it next week.

    For the past few days on and off, I've been knocking up a small Python script to scrape KingOfSat for all data required for a CCcam.channelinfo file, with the aim of using it to automatically compile full and complete files on demand. You can tailor the packages that are scraped easily with a simple configuration file, setting your preferred CAID and provid's as you please, or simply using the full and complete config file provided to generate an all-singing all-dancing CCcam.channelinfo. At the moment it scrapes fantastically and without fault, but it needs some tidying and an option to build a new config file, based upon the packages currently listed on KoS. Things like CAID's and Provid's aren't carried on KoS, and would need to be filled in one time manually (if you're not using the config I'll be providing with it).

    This will initially be released as a standard Python script at first, and depends on BeautifulSoup and Mechanize. Soon after I'll be compiling it into both a pyinstaller package and standalone Windows executable to ease it's use. The intention for this isn't really that it'd be used on-receiver, as that would generate a fair bit of additional load for KoS and would also be impolite, but more so people can automate the creation of their own CCcam.channelinfo files for distribution/cron wget.

    If there's any functionality any of you would like to see from such a tool, here would be the place to put forward your ideas.

    Cheers.

  2. The Following 9 Users Say Thank You to slain For This Useful Post:

    bassethound (28-09-12),bertp (13-10-12),Delagroov (28-09-12),judge (28-09-12),Larry-G (28-09-12),Maxwell (28-09-12),Rob van der Does (28-09-12),Sicilian (28-09-12),tinbin300 (20-02-13)

  3. #2
    ViX Beta Tester Rob van der Does's Avatar

    Join Date
    Apr 2010
    Posts
    34,728
    Thanks
    1,558
    Thanked 9,228 Times in 6,511 Posts
    I'm looking forward to your tool. I have been struggling for quite some time to try to keep the files up-to-date.

    Still I think that a plugin with an 'auto-refresh'-functionality wouldn't be too bad?


    Help asked via PM will be ignored.
    The forum is there for help and all will benefit from your questions.
    NO CARD SHARING TALK WILL BE TOLERATED, LAN OR WAN, IN OPEN FORUM OR PM !

    English is not my native tongue.
    I apologise for all my grammar, spelling and idiom errors.

  4. #3
    Member

    Join Date
    Nov 2010
    Posts
    31
    Thanks
    10
    Thanked 24 Times in 10 Posts
    It would be nice to have an updater as a plugin I agree, but I think the more polite way to do it would be to generate the available packages of a channelinfo file once per day on a server, and have the plugin pull it from there. I'm not sure KoS would appreciate thousands of additional bot hits per day.

    I'll look at making the plugin as flexible as possible anyway, and leave how it's leveraged to the user.

  5. The Following 2 Users Say Thank You to slain For This Useful Post:

    bassethound (28-09-12),Rob van der Does (28-09-12)

  6. #4
    Member

    Join Date
    Nov 2010
    Posts
    31
    Thanks
    10
    Thanked 24 Times in 10 Posts
    I'd planned to have something released over the weekend, but a failed hard-drive, restoring backups and frantic attempts at some data recovery put an end to that. Fortunately I didn't lose my work on this so far! I'll have something for you all to play with soon.

  7. The Following User Says Thank You to slain For This Useful Post:

    Rob van der Does (01-10-12)

  8. #5
    Member

    Join Date
    Nov 2010
    Posts
    31
    Thanks
    10
    Thanked 24 Times in 10 Posts
    Ok, so here is the first release. You'll need Python 2.7, plus the BeautifulSoup and Mechanize libraries. Looking at the supplied config file you should be able to get the gist of what you need to do with it. I should have another release out in the next few days or so, with some new features and better error checking.
    Attached Files Attached Files

  9. The Following 3 Users Say Thank You to slain For This Useful Post:

    bassethound (04-10-12),Huevos (04-10-12),Larry-G (03-10-12)

  10. #6
    Moderator Huevos's Avatar

    Join Date
    Jun 2010
    Location
    38.5N, 0.5W
    Posts
    7,426
    Thanks
    1,539
    Thanked 2,667 Times in 1,783 Posts
    Slain, I know you are in a *NIX environment but I'm hoping you might be able to give me an idea where I am going wrong with my windows setup. I've installed BeautifulSoup but python can't find it. I've attached a command line printout of the install and the error.
    Attached Files Attached Files

  11. #7
    Member

    Join Date
    Nov 2010
    Posts
    31
    Thanks
    10
    Thanked 24 Times in 10 Posts
    Quote Originally Posted by Huevos View Post
    Slain, I know you are in a *NIX environment but I'm hoping you might be able to give me an idea where I am going wrong with my windows setup. I've installed BeautifulSoup but python can't find it. I've attached a command line printout of the install and the error.
    I'm running the following versions of both modules:

    BeautifulSoup - 3.2.0
    Mechanize - 0.2.5

    I see you're using BS4, which has been re-factored a great deal and isn't too compatible with BS3. Could you try removing BS4 and try BS3 instead?

  12. #8
    Member

    Join Date
    Nov 2010
    Posts
    31
    Thanks
    10
    Thanked 24 Times in 10 Posts
    Just to add to my last post, would you guys feel that an "enc" field in the config would be beneficial, for adding the type of encryption to the output ie.

    090F:000000:1F9A "Viasat - TV 4 Sweden [NDS3]"

    Currently this isn't in there but it'd be relatively simple to add it. I'd just need to add it based on whether the Encryption field is listed as "Clear" or not on KOS. It might not even be worth distinguishing between them, based on the fact that the clear channels wouldn't be hitting CCcam anyway. It'd only make the lines more accurate in a factual sense.

  13. #9
    Moderator Huevos's Avatar

    Join Date
    Jun 2010
    Location
    38.5N, 0.5W
    Posts
    7,426
    Thanks
    1,539
    Thanked 2,667 Times in 1,783 Posts
    Quote Originally Posted by slain View Post
    I'm running the following versions of both modules:

    BeautifulSoup - 3.2.0
    Mechanize - 0.2.5

    I see you're using BS4, which has been re-factored a great deal and isn't too compatible with BS3. Could you try removing BS4 and try BS3 instead?
    I'll swap to BS3 instead, but what directory should it be installed in as I think this is the problem.

  14. #10
    Member

    Join Date
    Nov 2010
    Posts
    31
    Thanks
    10
    Thanked 24 Times in 10 Posts
    Quote Originally Posted by Huevos View Post
    I'll swap to BS3 instead, but what directory should it be installed in as I think this is the problem.
    I really don't have much of a clue when it comes to Windows mate, but for what it's worth it did look to be installing in the correct place to me. Anyone else able to help Huevos?

    Just to add, I have a default kos_scrape.conf file here that I scraped that just needs correct CAID and provider ID's adding for each package. The default config generation will be built into the next release of the script, which will likely be tonight.

    Basically when someone can post back a completed config with the correct CAID's and provider ID's, we'll have a tool that can generate complete and up-to-date CCcam.channelinfo files instantly.
    Attached Files Attached Files

  15. #11
    Moderator Huevos's Avatar

    Join Date
    Jun 2010
    Location
    38.5N, 0.5W
    Posts
    7,426
    Thanks
    1,539
    Thanked 2,667 Times in 1,783 Posts
    Quote Originally Posted by slain View Post
    I really don't have much of a clue when it comes to Windows mate, but for what it's worth it did look to be installing in the correct place to me.
    It installed into the same directory as setup.py. I don't think that can be correct. It finds mechanize ok and I installed that the same way. I'll have to have a look where it installed that to and move BS3 to the same parent directory. Anyway I'll have another go tomorrow when I'm feeling a bit more awake.

  16. #12
    Member

    Join Date
    Nov 2010
    Posts
    31
    Thanks
    10
    Thanked 24 Times in 10 Posts
    Quote Originally Posted by Huevos View Post
    It installed into the same directory as setup.py. I don't think that can be correct. It finds mechanize ok and I installed that the same way. I'll have to have a look where it installed that to and move BS3 to the same parent directory. Anyway I'll have another go tomorrow when I'm feeling a bit more awake.
    No worries. Sorry I can't be of any more help on the Windows side of things. If you get really stuck you could use a virtual machine within VirtualBox for Ubuntu 12.04, which should make things a lot easier.

    Just to add to earlier posts, I'll be adding an update to use multiple CAID's for a package some time this weekend. Basically it'll mean that every channel gets added x times, x being the number of CAID's available for the package. The default config generation routine will be included in that release.

  17. #13
    Member

    Join Date
    Nov 2010
    Posts
    31
    Thanks
    10
    Thanked 24 Times in 10 Posts
    Ok, I've got the multiple card functionality working now, and clear channels aren't processed. The new format for the config is as follows:

    Code:
    [slydigital]
    name = Sly Digital
    url = http://en.kingofsat.net/pack-slydigital.php
    cards = 0963:000000
    If Sly were to come out with a new card with a CAID of FFFF and provider of 012345, the config for Sly would look like:

    Code:
    [slydigital]
    name = Sly Digital
    url = http://en.kingofsat.net/pack-slydigital.php
    cards = 0963:000000,FFFF:012345
    So yeah, it's a bit more useful now. As it currently stands, the tool does everything you'd want it to, probably bar listing encryption in the channelinfo lines (no big deal so far, really).

  18. The Following User Says Thank You to slain For This Useful Post:

    jones25 (14-10-12)

  19. #14
    Member

    Join Date
    Nov 2010
    Posts
    31
    Thanks
    10
    Thanked 24 Times in 10 Posts

    Cool

    Ok, I've finished the second version of the script now. New features include:

    * Multiple CAID/providers supported
    * Ability to create a default config file ready to edit as you please
    * Better output to the terminal (tells you exactly what it's doing)
    * Ability to disable a package in the config file, ie for un-needed packages
    * Don't process FTA channels any more (pointless and incorrect)

    That's all I can think of right now. Basically I think everything is there that is needed to create a pretty rockin' CCcam.channelinfo with very little effort. As I've said before, once someone creates a complete kos_scrape.conf file with correct card data in it'll generate a full file in a minute or two. I started off a kos_scrape.conf in the tarball attached, but haven't had time to complete it thus far.

    Please, do try your best to give me as much feedback as possible. I'll help where ever possible with folk using Linux, and hopefully someone will come along that can assist with the Windows side of things!

    Cheers,

    slain
    Attached Files Attached Files

  20. The Following 2 Users Say Thank You to slain For This Useful Post:

    bassethound (09-10-12),Larry-G (10-10-12)

  21. #15
    Senior Member

    Join Date
    Jun 2011
    Posts
    291
    Thanks
    34
    Thanked 18 Times in 16 Posts
    Is it possible to have channel numbers?

    Sent from my Galaxy Nexus using Tapatalk

Page 1 of 3 123 LastLast

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •